2023-12-25T20:36:42.701 INFO:root:teuthology version: 0.0.1.dev226+g9680d2a 2023-12-25T20:36:42.701 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/one workloads/rados_mon_workunits} --archive /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500952 --name yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500952/orig.config.yaml 2023-12-25T20:36:42.735 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-25T20:36:42.824 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500952 branch: wip-yuri5-testing-2023-12-15-0747-pacific description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/one workloads/rados_mon_workunits} email: yweinste@redhat.com first_in_suite: false job_id: '7500952' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri5-testing-2023-12-15-0747-pacific ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - but it is still running - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(AUTH_BAD_CAPS\) sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri5-testing-2023-12-15-0747-pacific sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 2834 sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri5-testing-2023-12-15-0747-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 targets: smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHwsX+5nIByVAfAQ8I7l1bnFaNLiDHwJYt5U5Zvs8zKhl42cGlAb4liT5eIqnXZhTjMK80xEHH1sp+lHie9P4w4= smithi191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD+ISzGpHk1nnk50JTFdMJHaU3XrYxVp8J6UrAV6J4AP0mz5orSF5TRpCHUvF8HD2drq6QTywF6azXOm3Jow5dE= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 20 thrash_delay: 1 - workunit: clients: client.0: - mon/pool_ops.sh - mon/crush_ops.sh - mon/osd.sh - mon/caps.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9680d2aba1e76f64ade295c81599c97907457848 timestamp: 2023-12-25_17:28:36 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 2023-12-25T20:36:42.825 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa; will attempt to use it 2023-12-25T20:36:42.825 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks 2023-12-25T20:36:42.825 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-25T20:36:42.827 INFO:teuthology.task.internal:Checking packages... 2023-12-25T20:36:42.854 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2' 2023-12-25T20:36:42.854 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-25T20:36:42.854 INFO:teuthology.packaging:ref: None 2023-12-25T20:36:42.854 INFO:teuthology.packaging:tag: None 2023-12-25T20:36:42.854 INFO:teuthology.packaging:branch: wip-yuri5-testing-2023-12-15-0747-pacific 2023-12-25T20:36:42.854 INFO:teuthology.packaging:sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 2023-12-25T20:36:42.854 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri5-testing-2023-12-15-0747-pacific 2023-12-25T20:36:43.097 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-385.gb14a0a10 2023-12-25T20:36:43.099 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-25T20:36:43.108 INFO:teuthology.task.internal:no buildpackages task found 2023-12-25T20:36:43.108 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-25T20:36:43.142 INFO:teuthology.task.internal:Saving configuration 2023-12-25T20:36:43.156 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-25T20:36:43.167 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-25T20:36:43.193 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500952', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-25 20:31:35.172889', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHwsX+5nIByVAfAQ8I7l1bnFaNLiDHwJYt5U5Zvs8zKhl42cGlAb4liT5eIqnXZhTjMK80xEHH1sp+lHie9P4w4='} 2023-12-25T20:36:43.218 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500952', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-25 20:31:35.174303', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD+ISzGpHk1nnk50JTFdMJHaU3XrYxVp8J6UrAV6J4AP0mz5orSF5TRpCHUvF8HD2drq6QTywF6azXOm3Jow5dE='} 2023-12-25T20:36:43.218 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-25T20:36:43.227 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2023-12-25T20:36:43.228 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2023-12-25T20:36:43.229 INFO:teuthology.run_tasks:Running task console_log... 2023-12-25T20:36:43.328 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f46f3a7de50>, signals=[15]) 2023-12-25T20:36:43.329 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-25T20:36:43.338 INFO:teuthology.task.internal:Opening connections... 2023-12-25T20:36:43.338 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2023-12-25T20:36:43.340 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T20:36:43.410 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2023-12-25T20:36:43.411 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T20:36:43.479 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-25T20:36:43.492 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2023-12-25T20:36:43.515 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2023-12-25T20:36:43.516 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2023-12-25T20:36:43.577 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2023-12-25T20:36:43.577 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="8" 2023-12-25T20:36:43.577 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2023-12-25T20:36:43.577 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2023-12-25T20:36:43.577 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="8" 2023-12-25T20:36:43.578 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el8" 2023-12-25T20:36:43.578 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-25T20:36:43.578 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2023-12-25T20:36:43.580 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-25T20:36:43.580 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2023-12-25T20:36:43.580 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-25T20:36:43.580 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-25T20:36:43.580 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-25T20:36:43.581 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2023-12-25T20:36:43.605 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2023-12-25T20:36:43.628 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2023-12-25T20:36:43.628 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2023-12-25T20:36:43.690 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="CentOS Stream" 2023-12-25T20:36:43.690 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="8" 2023-12-25T20:36:43.690 INFO:teuthology.orchestra.run.smithi191.stdout:ID="centos" 2023-12-25T20:36:43.690 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="rhel fedora" 2023-12-25T20:36:43.690 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="8" 2023-12-25T20:36:43.690 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el8" 2023-12-25T20:36:43.690 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-25T20:36:43.690 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2023-12-25T20:36:43.690 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-25T20:36:43.690 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://centos.org/" 2023-12-25T20:36:43.690 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-25T20:36:43.690 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-25T20:36:43.691 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-25T20:36:43.691 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2023-12-25T20:36:43.714 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-25T20:36:43.729 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-25T20:36:43.750 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-25T20:36:43.750 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2023-12-25T20:36:43.753 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2023-12-25T20:36:43.774 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-25T20:36:43.781 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-25T20:36:43.781 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2023-12-25T20:36:43.814 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2023-12-25T20:36:43.858 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-25T20:36:43.916 INFO:teuthology.run_tasks:Running task kernel... 2023-12-25T20:36:44.001 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-25T20:36:44.001 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-25T20:36:44.002 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-25T20:36:44.002 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-25T20:36:44.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T20:36:44.024 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2023-12-25T20:36:44.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T20:36:44.025 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2023-12-25T20:36:44.085 INFO:teuthology.orchestra.run.smithi153.stdout:4.18.0-489.el8.x86_64 2023-12-25T20:36:44.085 INFO:teuthology.task.kernel:Running kernel on smithi153: 4.18.0-489.el8.x86_64 2023-12-25T20:36:44.085 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum install -y kernel 2023-12-25T20:36:44.086 INFO:teuthology.orchestra.run.smithi191.stdout:4.18.0-489.el8.x86_64 2023-12-25T20:36:44.086 INFO:teuthology.task.kernel:Running kernel on smithi191: 4.18.0-489.el8.x86_64 2023-12-25T20:36:44.087 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum install -y kernel 2023-12-25T20:36:48.740 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - AppStream 16 MB/s | 35 MB 00:02 2023-12-25T20:36:51.655 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 8 - AppStream 7.1 MB/s | 35 MB 00:04 2023-12-25T20:37:11.471 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - BaseOS 3.5 MB/s | 57 MB 00:16 2023-12-25T20:37:18.445 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 8 - BaseOS 2.8 MB/s | 57 MB 00:20 2023-12-25T20:37:20.004 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - Extras 51 kB/s | 18 kB 00:00 2023-12-25T20:37:20.738 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - Extras common packages 12 kB/s | 6.9 kB 00:00 2023-12-25T20:37:21.015 INFO:teuthology.orchestra.run.smithi153.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-12-25T20:37:26.686 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 8 - Extras 80 kB/s | 18 kB 00:00 2023-12-25T20:37:27.354 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 8 - Extras common packages 14 kB/s | 6.9 kB 00:00 2023-12-25T20:37:27.647 INFO:teuthology.orchestra.run.smithi191.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-12-25T20:37:29.323 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2023-12-25T20:37:31.872 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 1.5 MB/s | 16 MB 00:10 2023-12-25T20:37:33.123 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 448 kB/s | 24 kB 00:00 2023-12-25T20:37:35.747 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 370 kB/s | 24 kB 00:00 2023-12-25T20:37:36.784 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-25T20:37:36.784 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-25T20:37:37.066 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:37:37.066 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:37:37.066 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2023-12-25T20:37:37.066 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:37:37.067 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-12-25T20:37:37.067 INFO:teuthology.orchestra.run.smithi191.stdout: kernel x86_64 4.18.0-529.el8 baseos 10 M 2023-12-25T20:37:37.067 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2023-12-25T20:37:37.067 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core x86_64 4.18.0-529.el8 baseos 43 M 2023-12-25T20:37:37.067 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules x86_64 4.18.0-529.el8 baseos 36 M 2023-12-25T20:37:37.067 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:37:37.067 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-12-25T20:37:37.067 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:37:37.068 INFO:teuthology.orchestra.run.smithi191.stdout:Install 3 Packages 2023-12-25T20:37:37.068 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:37:37.068 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 89 M 2023-12-25T20:37:37.069 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 96 M 2023-12-25T20:37:37.069 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-12-25T20:37:38.925 INFO:teuthology.orchestra.run.smithi191.stdout:(1/3): kernel-4.18.0-529.el8.x86_64.rpm 5.6 MB/s | 10 MB 00:01 2023-12-25T20:37:39.430 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-25T20:37:39.430 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-25T20:37:39.744 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:37:39.745 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:37:39.745 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2023-12-25T20:37:39.745 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:37:39.745 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-12-25T20:37:39.745 INFO:teuthology.orchestra.run.smithi153.stdout: kernel x86_64 4.18.0-529.el8 baseos 10 M 2023-12-25T20:37:39.745 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2023-12-25T20:37:39.746 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core x86_64 4.18.0-529.el8 baseos 43 M 2023-12-25T20:37:39.746 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules x86_64 4.18.0-529.el8 baseos 36 M 2023-12-25T20:37:39.746 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:37:39.746 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-12-25T20:37:39.746 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:37:39.746 INFO:teuthology.orchestra.run.smithi153.stdout:Install 3 Packages 2023-12-25T20:37:39.746 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:37:39.747 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 89 M 2023-12-25T20:37:39.747 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 96 M 2023-12-25T20:37:39.747 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-12-25T20:37:40.275 INFO:teuthology.orchestra.run.smithi191.stdout:(2/3): kernel-modules-4.18.0-529.el8.x86_64.rpm 11 MB/s | 36 MB 00:03 2023-12-25T20:37:40.934 INFO:teuthology.orchestra.run.smithi191.stdout:(3/3): kernel-core-4.18.0-529.el8.x86_64.rpm 11 MB/s | 43 MB 00:03 2023-12-25T20:37:40.934 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:37:40.934 INFO:teuthology.orchestra.run.smithi191.stdout:Total 23 MB/s | 89 MB 00:03 2023-12-25T20:37:41.614 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-12-25T20:37:41.907 INFO:teuthology.orchestra.run.smithi153.stdout:(1/3): kernel-4.18.0-529.el8.x86_64.rpm 5.2 MB/s | 10 MB 00:01 2023-12-25T20:37:42.048 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-12-25T20:37:42.048 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-12-25T20:37:43.098 INFO:teuthology.orchestra.run.smithi153.stdout:(2/3): kernel-modules-4.18.0-529.el8.x86_64.rpm 11 MB/s | 36 MB 00:03 2023-12-25T20:37:43.286 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-12-25T20:37:43.307 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-12-25T20:37:45.990 INFO:teuthology.orchestra.run.smithi153.stdout:(3/3): kernel-core-4.18.0-529.el8.x86_64.rpm 7.1 MB/s | 43 MB 00:06 2023-12-25T20:37:45.991 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:37:45.991 INFO:teuthology.orchestra.run.smithi153.stdout:Total 14 MB/s | 89 MB 00:06 2023-12-25T20:37:46.079 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-12-25T20:37:46.607 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-25T20:37:46.666 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-12-25T20:37:47.099 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-12-25T20:37:47.100 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-12-25T20:37:48.323 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-12-25T20:37:48.356 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-12-25T20:37:50.134 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-25T20:37:50.573 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-25T20:37:51.144 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-12-25T20:37:52.006 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-25T20:37:54.001 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-25T20:37:54.330 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-4.18.0-529.el8.x86_64 3/3 2023-12-25T20:37:55.610 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-25T20:37:56.005 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-25T20:37:59.442 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-25T20:37:59.736 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-4.18.0-529.el8.x86_64 3/3 2023-12-25T20:38:16.341 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 3/3 2023-12-25T20:38:16.366 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-25T20:38:16.727 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-4.18.0-529.el8.x86_64 3/3 2023-12-25T20:38:16.727 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-4.18.0-529.el8.x86_64 1/3 2023-12-25T20:38:16.727 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-core-4.18.0-529.el8.x86_64 2/3 2023-12-25T20:38:17.499 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-25T20:38:17.499 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:38:17.499 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-12-25T20:38:17.500 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-4.18.0-529.el8.x86_64 kernel-core-4.18.0-529.el8.x86_64 2023-12-25T20:38:17.500 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-4.18.0-529.el8.x86_64 2023-12-25T20:38:17.500 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:38:17.500 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:38:17.642 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2023-12-25T20:38:17.861 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-25T20:38:17.862 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-25T20:38:17.862 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-529.el8.x86_64 2023-12-25T20:38:17.862 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-25T20:38:17.862 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-25T20:38:17.862 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-25T20:38:17.862 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi191.front.sepia.ceph.com, path=None, version=distro) 2023-12-25T20:38:17.862 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2023-12-25T20:38:18.071 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-25T20:38:18.071 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-25T20:38:18.071 DEBUG:teuthology.orchestra.run.smithi191:> sudo rpm -qi grub2-tools 2023-12-25T20:38:18.105 INFO:teuthology.orchestra.run.smithi191.stdout:Name : grub2-tools 2023-12-25T20:38:18.105 INFO:teuthology.orchestra.run.smithi191.stdout:Epoch : 1 2023-12-25T20:38:18.105 INFO:teuthology.orchestra.run.smithi191.stdout:Version : 2.02 2023-12-25T20:38:18.105 INFO:teuthology.orchestra.run.smithi191.stdout:Release : 129.el8 2023-12-25T20:38:18.105 INFO:teuthology.orchestra.run.smithi191.stdout:Architecture: x86_64 2023-12-25T20:38:18.105 INFO:teuthology.orchestra.run.smithi191.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-25T20:38:18.105 INFO:teuthology.orchestra.run.smithi191.stdout:Group : System Environment/Base 2023-12-25T20:38:18.105 INFO:teuthology.orchestra.run.smithi191.stdout:Size : 9265586 2023-12-25T20:38:18.105 INFO:teuthology.orchestra.run.smithi191.stdout:License : GPLv3+ 2023-12-25T20:38:18.105 INFO:teuthology.orchestra.run.smithi191.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-25T20:38:18.105 INFO:teuthology.orchestra.run.smithi191.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-25T20:38:18.106 INFO:teuthology.orchestra.run.smithi191.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-25T20:38:18.106 INFO:teuthology.orchestra.run.smithi191.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-25T20:38:18.106 INFO:teuthology.orchestra.run.smithi191.stdout:Relocations : (not relocatable) 2023-12-25T20:38:18.106 INFO:teuthology.orchestra.run.smithi191.stdout:Packager : CentOS BuildSystem 2023-12-25T20:38:18.106 INFO:teuthology.orchestra.run.smithi191.stdout:Vendor : CentOS 2023-12-25T20:38:18.106 INFO:teuthology.orchestra.run.smithi191.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-25T20:38:18.106 INFO:teuthology.orchestra.run.smithi191.stdout:Summary : Support tools for GRUB. 2023-12-25T20:38:18.106 INFO:teuthology.orchestra.run.smithi191.stdout:Description : 2023-12-25T20:38:18.106 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:38:18.106 INFO:teuthology.orchestra.run.smithi191.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-25T20:38:18.106 INFO:teuthology.orchestra.run.smithi191.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-25T20:38:18.106 INFO:teuthology.orchestra.run.smithi191.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-25T20:38:18.106 INFO:teuthology.orchestra.run.smithi191.stdout:hardware devices. 2023-12-25T20:38:18.106 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:38:18.106 INFO:teuthology.orchestra.run.smithi191.stdout:This subpackage provides tools for support of all platforms. 2023-12-25T20:38:18.107 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-25T20:38:18.107 INFO:teuthology.task.kernel:Updating grub on smithi191 to boot 4.18.0-529.el8.x86_64 2023-12-25T20:38:18.108 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-25T20:38:19.679 INFO:teuthology.orchestra.run.smithi191.stderr:Generating grub configuration file ... 2023-12-25T20:38:21.536 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 3/3 2023-12-25T20:38:21.560 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-25T20:38:21.896 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-4.18.0-529.el8.x86_64 3/3 2023-12-25T20:38:21.896 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-4.18.0-529.el8.x86_64 1/3 2023-12-25T20:38:21.896 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-core-4.18.0-529.el8.x86_64 2/3 2023-12-25T20:38:21.929 INFO:teuthology.orchestra.run.smithi191.stderr:done 2023-12-25T20:38:21.930 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2023-12-25T20:38:21.965 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.Vs76eNNiF6 2023-12-25T20:38:21.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Vs76eNNiF6 2023-12-25T20:38:22.039 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.Vs76eNNiF6 2023-12-25T20:38:22.191 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.Vs76eNNiF6 is 6KB 2023-12-25T20:38:22.239 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.Vs76eNNiF6 2023-12-25T20:38:22.256 DEBUG:teuthology.orchestra.run.smithi191:> sudo /bin/ls /boot/loader/entries || true 2023-12-25T20:38:22.337 INFO:teuthology.orchestra.run.smithi191.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-25T20:38:22.338 INFO:teuthology.orchestra.run.smithi191.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-25T20:38:22.338 INFO:teuthology.orchestra.run.smithi191.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-25T20:38:22.338 INFO:teuthology.orchestra.run.smithi191.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64.conf 2023-12-25T20:38:22.339 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64 2023-12-25T20:38:22.472 DEBUG:teuthology.orchestra.run.smithi191:> sudo shutdown -r now 2023-12-25T20:38:22.631 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-25T20:38:22.679 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:38:22.679 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-12-25T20:38:22.679 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-4.18.0-529.el8.x86_64 kernel-core-4.18.0-529.el8.x86_64 2023-12-25T20:38:22.680 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-4.18.0-529.el8.x86_64 2023-12-25T20:38:22.680 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:38:22.680 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:38:22.772 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2023-12-25T20:38:23.028 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-25T20:38:23.029 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-25T20:38:23.029 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-529.el8.x86_64 2023-12-25T20:38:23.029 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-25T20:38:23.029 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-25T20:38:23.030 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-25T20:38:23.030 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2023-12-25T20:38:23.030 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2023-12-25T20:38:23.248 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-25T20:38:23.249 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-25T20:38:23.249 DEBUG:teuthology.orchestra.run.smithi153:> sudo rpm -qi grub2-tools 2023-12-25T20:38:23.282 INFO:teuthology.orchestra.run.smithi153.stdout:Name : grub2-tools 2023-12-25T20:38:23.282 INFO:teuthology.orchestra.run.smithi153.stdout:Epoch : 1 2023-12-25T20:38:23.282 INFO:teuthology.orchestra.run.smithi153.stdout:Version : 2.02 2023-12-25T20:38:23.283 INFO:teuthology.orchestra.run.smithi153.stdout:Release : 129.el8 2023-12-25T20:38:23.283 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: x86_64 2023-12-25T20:38:23.283 INFO:teuthology.orchestra.run.smithi153.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-25T20:38:23.283 INFO:teuthology.orchestra.run.smithi153.stdout:Group : System Environment/Base 2023-12-25T20:38:23.283 INFO:teuthology.orchestra.run.smithi153.stdout:Size : 9265586 2023-12-25T20:38:23.283 INFO:teuthology.orchestra.run.smithi153.stdout:License : GPLv3+ 2023-12-25T20:38:23.283 INFO:teuthology.orchestra.run.smithi153.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-25T20:38:23.283 INFO:teuthology.orchestra.run.smithi153.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-25T20:38:23.283 INFO:teuthology.orchestra.run.smithi153.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-25T20:38:23.283 INFO:teuthology.orchestra.run.smithi153.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-25T20:38:23.283 INFO:teuthology.orchestra.run.smithi153.stdout:Relocations : (not relocatable) 2023-12-25T20:38:23.283 INFO:teuthology.orchestra.run.smithi153.stdout:Packager : CentOS BuildSystem 2023-12-25T20:38:23.283 INFO:teuthology.orchestra.run.smithi153.stdout:Vendor : CentOS 2023-12-25T20:38:23.283 INFO:teuthology.orchestra.run.smithi153.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-25T20:38:23.283 INFO:teuthology.orchestra.run.smithi153.stdout:Summary : Support tools for GRUB. 2023-12-25T20:38:23.283 INFO:teuthology.orchestra.run.smithi153.stdout:Description : 2023-12-25T20:38:23.284 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:38:23.284 INFO:teuthology.orchestra.run.smithi153.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-25T20:38:23.284 INFO:teuthology.orchestra.run.smithi153.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-25T20:38:23.284 INFO:teuthology.orchestra.run.smithi153.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-25T20:38:23.284 INFO:teuthology.orchestra.run.smithi153.stdout:hardware devices. 2023-12-25T20:38:23.284 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:38:23.284 INFO:teuthology.orchestra.run.smithi153.stdout:This subpackage provides tools for support of all platforms. 2023-12-25T20:38:23.285 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-25T20:38:23.285 INFO:teuthology.task.kernel:Updating grub on smithi153 to boot 4.18.0-529.el8.x86_64 2023-12-25T20:38:23.285 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-25T20:38:24.698 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2023-12-25T20:38:26.586 INFO:teuthology.orchestra.run.smithi153.stderr:done 2023-12-25T20:38:26.588 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2023-12-25T20:38:26.623 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.m1HTEwQr24 2023-12-25T20:38:26.623 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.m1HTEwQr24 2023-12-25T20:38:26.697 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.m1HTEwQr24 2023-12-25T20:38:26.843 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.m1HTEwQr24 is 6KB 2023-12-25T20:38:26.890 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.m1HTEwQr24 2023-12-25T20:38:26.907 DEBUG:teuthology.orchestra.run.smithi153:> sudo /bin/ls /boot/loader/entries || true 2023-12-25T20:38:26.993 INFO:teuthology.orchestra.run.smithi153.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-25T20:38:26.993 INFO:teuthology.orchestra.run.smithi153.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-25T20:38:26.993 INFO:teuthology.orchestra.run.smithi153.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-25T20:38:26.993 INFO:teuthology.orchestra.run.smithi153.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64.conf 2023-12-25T20:38:26.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64 2023-12-25T20:38:27.122 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2023-12-25T20:38:52.497 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-25T20:38:52.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-12-25T20:38:52.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T20:38:57.129 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-25T20:38:57.130 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-12-25T20:38:57.130 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T20:39:31.602 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2023-12-25T20:39:40.609 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-12-25T20:39:40.610 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T20:39:52.509 DEBUG:teuthology.orchestra.remote:timed out 2023-12-25T20:39:58.997 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2023-12-25T20:40:01.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-12-25T20:40:01.514 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T20:40:01.920 DEBUG:teuthology.orchestra.run.smithi191:> true 2023-12-25T20:40:02.198 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-12-25T20:40:02.198 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-529.el8.x86_64"... 2023-12-25T20:40:02.198 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2023-12-25T20:40:02.255 INFO:teuthology.orchestra.run.smithi191.stdout:4.18.0-529.el8.x86_64 2023-12-25T20:40:02.256 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-529.el8.x86_64 vs 4.18.0-529.el8.x86_64 2023-12-25T20:40:02.256 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-25T20:40:02.256 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-25T20:40:03.257 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-25T20:40:03.257 DEBUG:teuthology.orchestra.run.smithi191:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-25T20:40:03.346 INFO:teuthology.orchestra.run.smithi191.stdout:ttyS1 2023-12-25T20:40:03.358 DEBUG:teuthology.parallel:result is None 2023-12-25T20:40:10.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-12-25T20:40:10.999 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T20:40:11.409 DEBUG:teuthology.orchestra.run.smithi153:> true 2023-12-25T20:40:11.758 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-12-25T20:40:11.758 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-529.el8.x86_64"... 2023-12-25T20:40:11.758 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2023-12-25T20:40:11.775 INFO:teuthology.orchestra.run.smithi153.stdout:4.18.0-529.el8.x86_64 2023-12-25T20:40:11.775 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-529.el8.x86_64 vs 4.18.0-529.el8.x86_64 2023-12-25T20:40:11.775 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-25T20:40:11.775 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-25T20:40:12.776 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-25T20:40:12.776 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-25T20:40:12.860 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2023-12-25T20:40:12.888 DEBUG:teuthology.parallel:result is None 2023-12-25T20:40:12.889 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-25T20:40:12.896 INFO:teuthology.task.internal:Creating test directory... 2023-12-25T20:40:12.897 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-25T20:40:12.900 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-25T20:40:12.925 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-25T20:40:12.959 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-25T20:40:12.991 INFO:teuthology.task.internal:Creating archive directory... 2023-12-25T20:40:12.991 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-25T20:40:12.994 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-25T20:40:13.031 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-25T20:40:13.038 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-25T20:40:13.038 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-25T20:40:13.057 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-25T20:40:13.105 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-25T20:40:13.116 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-25T20:40:13.120 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-25T20:40:13.130 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-25T20:40:13.132 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-25T20:40:13.139 INFO:teuthology.task.internal:Configuring sudo... 2023-12-25T20:40:13.139 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-25T20:40:13.141 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-25T20:40:13.188 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-25T20:40:13.197 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-25T20:40:13.197 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-25T20:40:13.214 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-25T20:40:13.247 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-25T20:40:13.319 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-25T20:40:13.375 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-25T20:40:13.464 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-25T20:40:13.510 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-12-25T20:40:13.510 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-25T20:40:13.579 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-25T20:40:13.627 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-25T20:40:13.675 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-25T20:40:13.763 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-25T20:40:13.804 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-25T20:40:13.805 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-25T20:40:13.872 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2023-12-25T20:40:13.873 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2023-12-25T20:40:13.956 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-25T20:40:13.981 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-25T20:40:14.335 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-25T20:40:14.343 INFO:teuthology.task.internal:Starting timer... 2023-12-25T20:40:14.344 INFO:teuthology.run_tasks:Running task pcp... 2023-12-25T20:40:14.375 INFO:teuthology.run_tasks:Running task selinux... 2023-12-25T20:40:14.403 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-25T20:40:14.404 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2023-12-25T20:40:14.521 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2023-12-25T20:40:14.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2023-12-25T20:40:14.601 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: 2023-12-25T20:40:14.603 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-25T20:40:14.603 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2023-12-25T20:40:14.649 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2023-12-25T20:40:14.649 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2023-12-25T20:40:14.712 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2023-12-25T20:40:14.712 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi153.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi191.front.sepia.ceph.com': 'permissive'} 2023-12-25T20:40:14.712 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-25T20:40:14.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T20:40:14.739 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-25T20:40:14.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T20:40:14.767 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-25T20:40:14.767 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2023-12-25T20:40:14.873 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2023-12-25T20:40:14.899 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-25T20:40:14.909 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-25T20:40:14.931 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-25T20:40:14.932 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi153.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-25T20:47:44.430 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi153.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2023-12-25T20:47:44.431 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-12-25T20:47:44.433 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T20:47:44.502 DEBUG:teuthology.orchestra.run.smithi153:> true 2023-12-25T20:47:44.567 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-12-25T20:47:44.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-12-25T20:47:44.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T20:47:44.637 DEBUG:teuthology.orchestra.run.smithi191:> true 2023-12-25T20:47:44.712 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-12-25T20:47:44.713 INFO:teuthology.run_tasks:Running task clock... 2023-12-25T20:47:44.724 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-25T20:47:44.724 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-25T20:47:44.724 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-25T20:47:44.727 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-25T20:47:44.727 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-25T20:47:44.756 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-25T20:47:44.769 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-25T20:47:44.786 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-25T20:47:44.801 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-25T20:47:44.837 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2023-12-25T20:47:44.848 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2023-12-25T20:47:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-25T20:47:44.865 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2023-12-25T20:47:44.874 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-25T20:47:44.877 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2023-12-25T20:47:44.892 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-25T20:47:44.905 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-25T20:47:44.940 INFO:teuthology.orchestra.run.smithi153.stderr:bash: ntpq: command not found 2023-12-25T20:47:44.944 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-25T20:47:44.944 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2023-12-25T20:47:44.945 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T20:47:44.945 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T20:47:44.945 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T20:47:44.945 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T20:47:44.970 INFO:teuthology.orchestra.run.smithi191.stderr:bash: ntpq: command not found 2023-12-25T20:47:44.973 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-25T20:47:44.973 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2023-12-25T20:47:44.974 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T20:47:44.974 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T20:47:44.974 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T20:47:44.974 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T20:47:44.975 INFO:teuthology.run_tasks:Running task install... 2023-12-25T20:47:44.983 DEBUG:teuthology.task.install:project ceph 2023-12-25T20:47:44.983 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2'}} 2023-12-25T20:47:44.983 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2'} 2023-12-25T20:47:44.983 INFO:teuthology.task.install:Using flavor: default 2023-12-25T20:47:44.989 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-25T20:47:44.989 INFO:teuthology.task.install:extra packages: [] 2023-12-25T20:47:44.989 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2', 'tag': None, 'wait_for_package': False} 2023-12-25T20:47:44.990 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 2023-12-25T20:47:44.992 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2', 'tag': None, 'wait_for_package': False} 2023-12-25T20:47:44.992 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 2023-12-25T20:47:45.210 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-15-0747-pacific/b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/centos/8/flavors/default/ 2023-12-25T20:47:45.210 INFO:teuthology.task.install.rpm:Package version is 16.2.14-385.gb14a0a10 2023-12-25T20:47:45.216 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-15-0747-pacific/b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/centos/8/flavors/default/ 2023-12-25T20:47:45.216 INFO:teuthology.task.install.rpm:Package version is 16.2.14-385.gb14a0a10 2023-12-25T20:47:45.344 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-15-0747-pacific/b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-15-0747-pacific/b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-15-0747-pacific/b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-25T20:47:45.345 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-25T20:47:45.345 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-25T20:47:45.346 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-15-0747-pacific/b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-15-0747-pacific/b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-15-0747-pacific/b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-25T20:47:45.346 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-12-25T20:47:45.346 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-25T20:47:45.374 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-25T20:47:45.375 DEBUG:teuthology.orchestra.run.smithi191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-25T20:47:45.376 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-25T20:47:45.376 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-25T20:47:45.444 DEBUG:teuthology.orchestra.run.smithi191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-25T20:47:45.445 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-25T20:47:45.523 DEBUG:teuthology.orchestra.run.smithi191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-25T20:47:45.523 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-25T20:47:45.590 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2023-12-25T20:47:45.590 INFO:teuthology.orchestra.run.smithi191.stdout:check_obsoletes = 1 2023-12-25T20:47:45.591 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2023-12-25T20:47:45.592 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum clean all 2023-12-25T20:47:45.857 INFO:teuthology.orchestra.run.smithi191.stdout:76 files removed 2023-12-25T20:47:45.858 INFO:teuthology.orchestra.run.smithi153.stdout:76 files removed 2023-12-25T20:47:45.885 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-radosgw 2023-12-25T20:47:45.886 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw 2023-12-25T20:47:46.905 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 124 kB/s | 76 kB 00:00 2023-12-25T20:47:46.925 INFO:teuthology.orchestra.run.smithi191.stdout:ceph packages for x86_64 122 kB/s | 76 kB 00:00 2023-12-25T20:47:47.280 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 46 kB/s | 15 kB 00:00 2023-12-25T20:47:47.368 INFO:teuthology.orchestra.run.smithi191.stdout:ceph noarch packages 38 kB/s | 15 kB 00:00 2023-12-25T20:47:47.798 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 2.9 kB/s | 1.4 kB 00:00 2023-12-25T20:47:47.935 INFO:teuthology.orchestra.run.smithi191.stdout:ceph source packages 2.7 kB/s | 1.4 kB 00:00 2023-12-25T20:47:49.913 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - AppStream 17 MB/s | 35 MB 00:02 2023-12-25T20:47:50.201 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 8 - AppStream 15 MB/s | 35 MB 00:02 2023-12-25T20:48:13.750 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 8 - BaseOS 3.3 MB/s | 57 MB 00:17 2023-12-25T20:48:14.713 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - BaseOS 3.1 MB/s | 57 MB 00:18 2023-12-25T20:48:22.383 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 8 - Extras 47 kB/s | 18 kB 00:00 2023-12-25T20:48:23.083 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.9 kB 00:00 2023-12-25T20:48:23.230 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - Extras 42 kB/s | 18 kB 00:00 2023-12-25T20:48:24.022 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - Extras common packages 11 kB/s | 6.9 kB 00:00 2023-12-25T20:48:24.476 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 8 - PowerTools 5.4 MB/s | 6.6 MB 00:01 2023-12-25T20:48:25.348 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - PowerTools 5.7 MB/s | 6.6 MB 00:01 2023-12-25T20:48:25.911 INFO:teuthology.orchestra.run.smithi191.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-12-25T20:48:26.783 INFO:teuthology.orchestra.run.smithi153.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-12-25T20:48:28.339 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 7.1 MB/s | 16 MB 00:02 2023-12-25T20:48:32.194 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 590 kB/s | 24 kB 00:00 2023-12-25T20:48:36.874 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:48:36.876 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:48:36.877 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2023-12-25T20:48:36.877 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:48:36.877 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-12-25T20:48:36.877 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 12 M 2023-12-25T20:48:36.877 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrading: 2023-12-25T20:48:36.877 INFO:teuthology.orchestra.run.smithi191.stdout: librados2 x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 3.7 M 2023-12-25T20:48:36.877 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1 x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 4.2 M 2023-12-25T20:48:36.877 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2023-12-25T20:48:36.877 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 6.1 M 2023-12-25T20:48:36.877 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 24 M 2023-12-25T20:48:36.877 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 26 k 2023-12-25T20:48:36.877 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-25T20:48:36.877 INFO:teuthology.orchestra.run.smithi191.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-25T20:48:36.878 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2 x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 825 k 2023-12-25T20:48:36.878 INFO:teuthology.orchestra.run.smithi191.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-25T20:48:36.878 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-12-25T20:48:36.878 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1 x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 493 k 2023-12-25T20:48:36.878 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2023-12-25T20:48:36.878 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2 x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 3.8 M 2023-12-25T20:48:36.878 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-25T20:48:36.878 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-12-25T20:48:36.878 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 45 k 2023-12-25T20:48:36.878 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 99 k 2023-12-25T20:48:36.878 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 215 k 2023-12-25T20:48:36.878 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-12-25T20:48:36.878 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 389 k 2023-12-25T20:48:36.878 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 369 k 2023-12-25T20:48:36.878 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 114 k 2023-12-25T20:48:36.879 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:48:36.879 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-12-25T20:48:36.879 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:48:36.879 INFO:teuthology.orchestra.run.smithi191.stdout:Install 21 Packages 2023-12-25T20:48:36.879 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrade 2 Packages 2023-12-25T20:48:36.879 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:48:36.879 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 58 M 2023-12-25T20:48:36.879 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-12-25T20:48:38.554 INFO:teuthology.orchestra.run.smithi191.stdout:(1/23): ceph-base-16.2.14-385.gb14a0a10.el8.x86 4.2 MB/s | 6.1 MB 00:01 2023-12-25T20:48:38.654 INFO:teuthology.orchestra.run.smithi191.stdout:(2/23): ceph-selinux-16.2.14-385.gb14a0a10.el8. 257 kB/s | 26 kB 00:00 2023-12-25T20:48:38.912 INFO:teuthology.orchestra.run.smithi191.stdout:(3/23): libcephfs2-16.2.14-385.gb14a0a10.el8.x8 3.1 MB/s | 825 kB 00:00 2023-12-25T20:48:39.563 INFO:teuthology.orchestra.run.smithi191.stdout:(4/23): ceph-radosgw-16.2.14-385.gb14a0a10.el8. 4.7 MB/s | 12 MB 00:02 2023-12-25T20:48:39.588 INFO:teuthology.orchestra.run.smithi191.stdout:(5/23): libradosstriper1-16.2.14-385.gb14a0a10. 731 kB/s | 493 kB 00:00 2023-12-25T20:48:39.722 INFO:teuthology.orchestra.run.smithi191.stdout:(6/23): python3-ceph-argparse-16.2.14-385.gb14a 337 kB/s | 45 kB 00:00 2023-12-25T20:48:40.197 INFO:teuthology.orchestra.run.smithi191.stdout:(7/23): python3-ceph-common-16.2.14-385.gb14a0a 208 kB/s | 99 kB 00:00 2023-12-25T20:48:40.264 INFO:teuthology.orchestra.run.smithi191.stdout:(8/23): librgw2-16.2.14-385.gb14a0a10.el8.x86_6 5.4 MB/s | 3.8 MB 00:00 2023-12-25T20:48:40.364 INFO:teuthology.orchestra.run.smithi191.stdout:(9/23): python3-cephfs-16.2.14-385.gb14a0a10.el 1.3 MB/s | 215 kB 00:00 2023-12-25T20:48:40.397 INFO:teuthology.orchestra.run.smithi191.stdout:(10/23): python3-rados-16.2.14-385.gb14a0a10.el 2.8 MB/s | 389 kB 00:00 2023-12-25T20:48:40.523 INFO:teuthology.orchestra.run.smithi191.stdout:(11/23): python3-rgw-16.2.14-385.gb14a0a10.el8. 915 kB/s | 114 kB 00:00 2023-12-25T20:48:40.615 INFO:teuthology.orchestra.run.smithi191.stdout:(12/23): python3-rbd-16.2.14-385.gb14a0a10.el8. 1.4 MB/s | 369 kB 00:00 2023-12-25T20:48:41.240 INFO:teuthology.orchestra.run.smithi191.stdout:(13/23): ceph-common-16.2.14-385.gb14a0a10.el8. 5.9 MB/s | 24 MB 00:04 2023-12-25T20:48:41.265 INFO:teuthology.orchestra.run.smithi191.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 399 kB/s | 259 kB 00:00 2023-12-25T20:48:41.291 INFO:teuthology.orchestra.run.smithi191.stdout:(15/23): librdkafka-1.6.1-1.el8.x86_64.rpm 882 kB/s | 677 kB 00:00 2023-12-25T20:48:41.407 INFO:teuthology.orchestra.run.smithi191.stdout:(16/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 331 kB/s | 47 kB 00:00 2023-12-25T20:48:41.558 INFO:teuthology.orchestra.run.smithi191.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 138 kB/s | 44 kB 00:00 2023-12-25T20:48:41.700 INFO:teuthology.orchestra.run.smithi191.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 749 kB/s | 306 kB 00:00 2023-12-25T20:48:41.767 INFO:teuthology.orchestra.run.smithi191.stdout:(19/23): leveldb-1.22-1.el8.x86_64.rpm 505 kB/s | 181 kB 00:00 2023-12-25T20:48:41.792 INFO:teuthology.orchestra.run.smithi191.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 818 kB/s | 75 kB 00:00 2023-12-25T20:48:41.817 INFO:teuthology.orchestra.run.smithi191.stdout:(21/23): liboath-2.6.2-3.el8.x86_64.rpm 227 kB/s | 59 kB 00:00 2023-12-25T20:48:42.326 INFO:teuthology.orchestra.run.smithi191.stdout:(22/23): librbd1-16.2.14-385.gb14a0a10.el8.x86_ 7.8 MB/s | 4.2 MB 00:00 2023-12-25T20:48:42.526 INFO:teuthology.orchestra.run.smithi191.stdout:(23/23): librados2-16.2.14-385.gb14a0a10.el8.x8 4.9 MB/s | 3.7 MB 00:00 2023-12-25T20:48:42.531 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:48:42.531 INFO:teuthology.orchestra.run.smithi191.stdout:Total 10 MB/s | 58 MB 00:05 2023-12-25T20:48:42.575 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-12-25T20:48:42.623 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-12-25T20:48:42.624 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-12-25T20:48:42.976 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-12-25T20:48:42.980 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-12-25T20:48:43.051 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 16 MB 00:16 2023-12-25T20:48:44.029 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-12-25T20:48:44.231 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-12-25T20:48:44.433 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-12-25T20:48:44.821 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-12-25T20:48:45.016 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librados2-2:16.2.14-385.gb14a0a10.el8.x86_64 3/25 2023-12-25T20:48:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:16.2.14-385.gb14a0a10.el8.x86_64 3/25 2023-12-25T20:48:45.414 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rados-2:16.2.14-385.gb14a0a10.el8.x86_64 4/25 2023-12-25T20:48:45.599 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs2-2:16.2.14-385.gb14a0a10.el8.x86_64 5/25 2023-12-25T20:48:45.990 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs2-2:16.2.14-385.gb14a0a10.el8.x86_64 5/25 2023-12-25T20:48:46.199 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librbd1-2:16.2.14-385.gb14a0a10.el8.x86_64 6/25 2023-12-25T20:48:46.247 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:16.2.14-385.gb14a0a10.el8.x86_64 6/25 2023-12-25T20:48:46.459 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/25 2023-12-25T20:48:46.898 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 364 kB/s | 24 kB 00:00 2023-12-25T20:48:46.954 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2023-12-25T20:48:47.115 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librgw2-2:16.2.14-385.gb14a0a10.el8.x86_64 9/25 2023-12-25T20:48:47.203 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librgw2-2:16.2.14-385.gb14a0a10.el8.x86_64 9/25 2023-12-25T20:48:47.396 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2023-12-25T20:48:47.546 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-argparse-2:16.2.14-385.gb14a0a10.el8. 11/25 2023-12-25T20:48:47.777 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cephfs-2:16.2.14-385.gb14a0a10.el8.x86_64 12/25 2023-12-25T20:48:48.017 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rgw-2:16.2.14-385.gb14a0a10.el8.x86_64 13/25 2023-12-25T20:48:48.249 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rbd-2:16.2.14-385.gb14a0a10.el8.x86_64 14/25 2023-12-25T20:48:48.407 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libradosstriper1-2:16.2.14-385.gb14a0a10.el8.x86_6 15/25 2023-12-25T20:48:48.476 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libradosstriper1-2:16.2.14-385.gb14a0a10.el8.x86_6 15/25 2023-12-25T20:48:48.676 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-12-25T20:48:48.874 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-12-25T20:48:49.113 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-12-25T20:48:49.308 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-common-2:16.2.14-385.gb14a0a10.el8.x8 19/25 2023-12-25T20:48:51.583 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:48:51.586 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:48:51.586 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2023-12-25T20:48:51.586 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:48:51.586 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-12-25T20:48:51.586 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 12 M 2023-12-25T20:48:51.586 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2023-12-25T20:48:51.586 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 3.7 M 2023-12-25T20:48:51.586 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 4.2 M 2023-12-25T20:48:51.586 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2023-12-25T20:48:51.586 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 6.1 M 2023-12-25T20:48:51.586 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 24 M 2023-12-25T20:48:51.587 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 26 k 2023-12-25T20:48:51.587 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-25T20:48:51.587 INFO:teuthology.orchestra.run.smithi153.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-25T20:48:51.587 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 825 k 2023-12-25T20:48:51.587 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-25T20:48:51.587 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-12-25T20:48:51.587 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 493 k 2023-12-25T20:48:51.587 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2023-12-25T20:48:51.587 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 3.8 M 2023-12-25T20:48:51.587 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-25T20:48:51.587 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-12-25T20:48:51.587 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 45 k 2023-12-25T20:48:51.587 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 99 k 2023-12-25T20:48:51.587 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 215 k 2023-12-25T20:48:51.588 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-12-25T20:48:51.588 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 389 k 2023-12-25T20:48:51.588 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 369 k 2023-12-25T20:48:51.588 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 114 k 2023-12-25T20:48:51.588 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:48:51.588 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-12-25T20:48:51.588 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:48:51.588 INFO:teuthology.orchestra.run.smithi153.stdout:Install 21 Packages 2023-12-25T20:48:51.588 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 2 Packages 2023-12-25T20:48:51.588 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:48:51.589 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 58 M 2023-12-25T20:48:51.589 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-12-25T20:48:51.820 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:16.2.14-385.gb14a0a10.el8.x86_64 20/25 2023-12-25T20:48:52.032 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-common-2:16.2.14-385.gb14a0a10.el8.x86_64 20/25 2023-12-25T20:48:52.727 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:16.2.14-385.gb14a0a10.el8.x86_64 20/25 2023-12-25T20:48:53.030 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-base-2:16.2.14-385.gb14a0a10.el8.x86_64 21/25 2023-12-25T20:48:53.132 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-base-2:16.2.14-385.gb14a0a10.el8.x86_64 21/25 2023-12-25T20:48:53.264 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-selinux-2:16.2.14-385.gb14a0a10.el8.x86_64 22/25 2023-12-25T20:48:55.574 INFO:teuthology.orchestra.run.smithi153.stdout:(1/23): ceph-common-16.2.14-385.gb14a0a10.el8.x 6.8 MB/s | 24 MB 00:03 2023-12-25T20:48:55.615 INFO:teuthology.orchestra.run.smithi153.stdout:(2/23): ceph-selinux-16.2.14-385.gb14a0a10.el8. 620 kB/s | 26 kB 00:00 2023-12-25T20:48:55.699 INFO:teuthology.orchestra.run.smithi153.stdout:(3/23): libcephfs2-16.2.14-385.gb14a0a10.el8.x8 9.7 MB/s | 825 kB 00:00 2023-12-25T20:48:55.775 INFO:teuthology.orchestra.run.smithi153.stdout:(4/23): libradosstriper1-16.2.14-385.gb14a0a10. 6.4 MB/s | 493 kB 00:00 2023-12-25T20:48:56.008 INFO:teuthology.orchestra.run.smithi153.stdout:(5/23): librgw2-16.2.14-385.gb14a0a10.el8.x86_6 16 MB/s | 3.8 MB 00:00 2023-12-25T20:48:56.050 INFO:teuthology.orchestra.run.smithi153.stdout:(6/23): python3-ceph-argparse-16.2.14-385.gb14a 1.1 MB/s | 45 kB 00:00 2023-12-25T20:48:56.100 INFO:teuthology.orchestra.run.smithi153.stdout:(7/23): python3-ceph-common-16.2.14-385.gb14a0a 1.9 MB/s | 99 kB 00:00 2023-12-25T20:48:56.151 INFO:teuthology.orchestra.run.smithi153.stdout:(8/23): python3-cephfs-16.2.14-385.gb14a0a10.el 4.2 MB/s | 215 kB 00:00 2023-12-25T20:48:56.210 INFO:teuthology.orchestra.run.smithi153.stdout:(9/23): python3-rados-16.2.14-385.gb14a0a10.el8 6.5 MB/s | 389 kB 00:00 2023-12-25T20:48:56.268 INFO:teuthology.orchestra.run.smithi153.stdout:(10/23): python3-rbd-16.2.14-385.gb14a0a10.el8. 6.2 MB/s | 369 kB 00:00 2023-12-25T20:48:56.318 INFO:teuthology.orchestra.run.smithi153.stdout:(11/23): python3-rgw-16.2.14-385.gb14a0a10.el8. 2.2 MB/s | 114 kB 00:00 2023-12-25T20:48:56.952 INFO:teuthology.orchestra.run.smithi153.stdout:(12/23): ceph-base-16.2.14-385.gb14a0a10.el8.x8 1.2 MB/s | 6.1 MB 00:04 2023-12-25T20:48:56.977 INFO:teuthology.orchestra.run.smithi153.stdout:(13/23): librdkafka-1.6.1-1.el8.x86_64.rpm 1.0 MB/s | 677 kB 00:00 2023-12-25T20:48:57.086 INFO:teuthology.orchestra.run.smithi153.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 405 kB/s | 44 kB 00:00 2023-12-25T20:48:57.261 INFO:teuthology.orchestra.run.smithi153.stdout:(15/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 268 kB/s | 47 kB 00:00 2023-12-25T20:48:57.461 INFO:teuthology.orchestra.run.smithi153.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 510 kB/s | 259 kB 00:00 2023-12-25T20:48:57.611 INFO:teuthology.orchestra.run.smithi153.stdout:(17/23): gperftools-libs-2.7-9.el8.x86_64.rpm 875 kB/s | 306 kB 00:00 2023-12-25T20:48:57.678 INFO:teuthology.orchestra.run.smithi153.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 881 kB/s | 59 kB 00:00 2023-12-25T20:48:57.746 INFO:teuthology.orchestra.run.smithi153.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2023-12-25T20:48:57.771 INFO:teuthology.orchestra.run.smithi153.stdout:(20/23): leveldb-1.22-1.el8.x86_64.rpm 585 kB/s | 181 kB 00:00 2023-12-25T20:48:59.346 INFO:teuthology.orchestra.run.smithi153.stdout:(21/23): librbd1-16.2.14-385.gb14a0a10.el8.x86_ 2.7 MB/s | 4.2 MB 00:01 2023-12-25T20:48:59.546 INFO:teuthology.orchestra.run.smithi153.stdout:(22/23): ceph-radosgw-16.2.14-385.gb14a0a10.el8 1.5 MB/s | 12 MB 00:07 2023-12-25T20:49:01.213 INFO:teuthology.orchestra.run.smithi153.stdout:(23/23): librados2-16.2.14-385.gb14a0a10.el8.x8 1.1 MB/s | 3.7 MB 00:03 2023-12-25T20:49:01.216 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:49:01.217 INFO:teuthology.orchestra.run.smithi153.stdout:Total 6.0 MB/s | 58 MB 00:09 2023-12-25T20:49:01.260 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-12-25T20:49:01.308 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-12-25T20:49:01.308 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-12-25T20:49:01.648 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-12-25T20:49:01.651 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-12-25T20:49:02.705 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-12-25T20:49:02.915 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-12-25T20:49:03.060 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-12-25T20:49:03.455 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-12-25T20:49:03.627 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:16.2.14-385.gb14a0a10.el8.x86_64 3/25 2023-12-25T20:49:03.704 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.14-385.gb14a0a10.el8.x86_64 3/25 2023-12-25T20:49:03.975 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:16.2.14-385.gb14a0a10.el8.x86_64 4/25 2023-12-25T20:49:04.135 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:16.2.14-385.gb14a0a10.el8.x86_64 5/25 2023-12-25T20:49:04.528 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:16.2.14-385.gb14a0a10.el8.x86_64 5/25 2023-12-25T20:49:04.727 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:16.2.14-385.gb14a0a10.el8.x86_64 6/25 2023-12-25T20:49:04.793 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.14-385.gb14a0a10.el8.x86_64 6/25 2023-12-25T20:49:05.019 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/25 2023-12-25T20:49:05.548 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2023-12-25T20:49:05.710 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:16.2.14-385.gb14a0a10.el8.x86_64 9/25 2023-12-25T20:49:05.791 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:16.2.14-385.gb14a0a10.el8.x86_64 9/25 2023-12-25T20:49:05.982 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2023-12-25T20:49:06.158 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:16.2.14-385.gb14a0a10.el8. 11/25 2023-12-25T20:49:06.346 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:16.2.14-385.gb14a0a10.el8.x86_64 12/25 2023-12-25T20:49:06.553 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:16.2.14-385.gb14a0a10.el8.x86_64 13/25 2023-12-25T20:49:06.743 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:16.2.14-385.gb14a0a10.el8.x86_64 14/25 2023-12-25T20:49:06.885 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:16.2.14-385.gb14a0a10.el8.x86_6 15/25 2023-12-25T20:49:06.938 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:16.2.14-385.gb14a0a10.el8.x86_6 15/25 2023-12-25T20:49:07.143 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-12-25T20:49:07.343 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-12-25T20:49:07.522 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-12-25T20:49:07.685 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:16.2.14-385.gb14a0a10.el8.x8 19/25 2023-12-25T20:49:10.198 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:16.2.14-385.gb14a0a10.el8.x86_64 20/25 2023-12-25T20:49:10.384 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:16.2.14-385.gb14a0a10.el8.x86_64 20/25 2023-12-25T20:49:11.100 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:16.2.14-385.gb14a0a10.el8.x86_64 20/25 2023-12-25T20:49:11.404 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:16.2.14-385.gb14a0a10.el8.x86_64 21/25 2023-12-25T20:49:11.472 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-selinux-2:16.2.14-385.gb14a0a10.el8.x86_64 22/25 2023-12-25T20:49:11.472 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /sys 2023-12-25T20:49:11.472 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /proc 2023-12-25T20:49:11.472 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /dev 2023-12-25T20:49:11.472 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /run 2023-12-25T20:49:11.472 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /mnt 2023-12-25T20:49:11.473 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /var/tmp 2023-12-25T20:49:11.473 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /home 2023-12-25T20:49:11.473 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /tmp 2023-12-25T20:49:11.473 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /dev 2023-12-25T20:49:11.473 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:49:11.506 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:16.2.14-385.gb14a0a10.el8.x86_64 21/25 2023-12-25T20:49:11.650 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:16.2.14-385.gb14a0a10.el8.x86_64 22/25 2023-12-25T20:49:12.677 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-radosgw-2:16.2.14-385.gb14a0a10.el8.x86_64 23/25 2023-12-25T20:49:12.788 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-385.gb14a0a10.el8.x86_64 23/25 2023-12-25T20:49:12.788 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-25T20:49:13.005 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-25T20:49:13.005 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-25T20:49:13.723 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-25T20:49:13.723 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-base-2:16.2.14-385.gb14a0a10.el8.x86_64 1/25 2023-12-25T20:49:13.723 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-common-2:16.2.14-385.gb14a0a10.el8.x86_64 2/25 2023-12-25T20:49:13.723 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-radosgw-2:16.2.14-385.gb14a0a10.el8.x86_64 3/25 2023-12-25T20:49:13.723 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-selinux-2:16.2.14-385.gb14a0a10.el8.x86_64 4/25 2023-12-25T20:49:13.723 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs2-2:16.2.14-385.gb14a0a10.el8.x86_64 5/25 2023-12-25T20:49:13.723 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libradosstriper1-2:16.2.14-385.gb14a0a10.el8.x86_6 6/25 2023-12-25T20:49:13.724 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librgw2-2:16.2.14-385.gb14a0a10.el8.x86_64 7/25 2023-12-25T20:49:13.724 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-argparse-2:16.2.14-385.gb14a0a10.el8. 8/25 2023-12-25T20:49:13.724 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-common-2:16.2.14-385.gb14a0a10.el8.x8 9/25 2023-12-25T20:49:13.724 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cephfs-2:16.2.14-385.gb14a0a10.el8.x86_64 10/25 2023-12-25T20:49:13.724 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rados-2:16.2.14-385.gb14a0a10.el8.x86_64 11/25 2023-12-25T20:49:13.724 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rbd-2:16.2.14-385.gb14a0a10.el8.x86_64 12/25 2023-12-25T20:49:13.724 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rgw-2:16.2.14-385.gb14a0a10.el8.x86_64 13/25 2023-12-25T20:49:13.724 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2023-12-25T20:49:13.724 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-12-25T20:49:13.724 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-12-25T20:49:13.724 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-12-25T20:49:13.724 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-12-25T20:49:13.724 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2023-12-25T20:49:13.725 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2023-12-25T20:49:13.725 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2023-12-25T20:49:13.725 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:16.2.14-385.gb14a0a10.el8.x86_64 22/25 2023-12-25T20:49:13.725 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-12-25T20:49:13.725 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:16.2.14-385.gb14a0a10.el8.x86_64 24/25 2023-12-25T20:49:14.959 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-12-25T20:49:14.959 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:49:14.959 INFO:teuthology.orchestra.run.smithi191.stdout:Upgraded: 2023-12-25T20:49:14.959 INFO:teuthology.orchestra.run.smithi191.stdout: librados2-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:14.959 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:14.959 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-12-25T20:49:14.959 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:14.959 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:14.959 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:14.959 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:14.959 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-25T20:49:14.959 INFO:teuthology.orchestra.run.smithi191.stdout: leveldb-1.22-1.el8.x86_64 2023-12-25T20:49:14.960 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:14.960 INFO:teuthology.orchestra.run.smithi191.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-25T20:49:14.960 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-12-25T20:49:14.960 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:14.960 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka-1.6.1-1.el8.x86_64 2023-12-25T20:49:14.960 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:14.960 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-25T20:49:14.960 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-25T20:49:14.960 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:14.960 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:14.960 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:14.960 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-25T20:49:14.960 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:14.960 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:14.960 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:14.961 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:49:14.961 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:49:15.128 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-test 2023-12-25T20:49:15.991 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:43 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:49:17.674 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:49:17.675 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:49:17.675 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2023-12-25T20:49:17.675 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:49:17.675 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-12-25T20:49:17.676 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 52 M 2023-12-25T20:49:17.676 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2023-12-25T20:49:17.676 INFO:teuthology.orchestra.run.smithi191.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-12-25T20:49:17.676 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 174 k 2023-12-25T20:49:17.676 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-12-25T20:49:17.676 INFO:teuthology.orchestra.run.smithi191.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-12-25T20:49:17.676 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-12-25T20:49:17.676 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:49:17.676 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-12-25T20:49:17.676 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:49:17.676 INFO:teuthology.orchestra.run.smithi191.stdout:Install 6 Packages 2023-12-25T20:49:17.676 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:49:17.677 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 53 M 2023-12-25T20:49:17.677 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 196 M 2023-12-25T20:49:17.677 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-12-25T20:49:18.011 INFO:teuthology.orchestra.run.smithi191.stdout:(1/6): libcephsqlite-16.2.14-385.gb14a0a10.el8. 593 kB/s | 174 kB 00:00 2023-12-25T20:49:18.186 INFO:teuthology.orchestra.run.smithi191.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 434 kB/s | 203 kB 00:00 2023-12-25T20:49:18.369 INFO:teuthology.orchestra.run.smithi191.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.7 MB/s | 323 kB 00:00 2023-12-25T20:49:18.395 INFO:teuthology.orchestra.run.smithi191.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 488 kB/s | 187 kB 00:00 2023-12-25T20:49:18.470 INFO:teuthology.orchestra.run.smithi191.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 705 kB/s | 71 kB 00:00 2023-12-25T20:49:20.637 INFO:teuthology.orchestra.run.smithi191.stdout:(6/6): ceph-test-16.2.14-385.gb14a0a10.el8.x86_ 18 MB/s | 52 MB 00:02 2023-12-25T20:49:20.638 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:49:20.639 INFO:teuthology.orchestra.run.smithi191.stdout:Total 18 MB/s | 53 MB 00:02 2023-12-25T20:49:20.681 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-12-25T20:49:20.702 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-12-25T20:49:20.702 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-12-25T20:49:21.030 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-12-25T20:49:21.033 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-12-25T20:49:21.510 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-12-25T20:49:21.737 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-12-25T20:49:21.970 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-12-25T20:49:22.117 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-25T20:49:22.191 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-25T20:49:22.380 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-12-25T20:49:22.534 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephsqlite-2:16.2.14-385.gb14a0a10.el8.x86_64 5/6 2023-12-25T20:49:27.153 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephsqlite-2:16.2.14-385.gb14a0a10.el8.x86_64 5/6 2023-12-25T20:49:27.407 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-test-2:16.2.14-385.gb14a0a10.el8.x86_64 6/6 2023-12-25T20:49:27.697 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-test-2:16.2.14-385.gb14a0a10.el8.x86_64 6/6 2023-12-25T20:49:27.698 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-test-2:16.2.14-385.gb14a0a10.el8.x86_64 1/6 2023-12-25T20:49:27.698 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephsqlite-2:16.2.14-385.gb14a0a10.el8.x86_64 2/6 2023-12-25T20:49:27.698 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-12-25T20:49:27.698 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-12-25T20:49:27.699 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-12-25T20:49:28.417 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-12-25T20:49:28.417 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:49:28.417 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-12-25T20:49:28.417 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:28.417 INFO:teuthology.orchestra.run.smithi191.stdout: jq-1.6-8.el8.x86_64 2023-12-25T20:49:28.417 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:28.417 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-12-25T20:49:28.418 INFO:teuthology.orchestra.run.smithi191.stdout: socat-1.7.4.1-1.el8.x86_64 2023-12-25T20:49:28.418 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-12-25T20:49:28.418 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:49:28.418 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:49:28.670 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph 2023-12-25T20:49:29.527 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:57 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:49:29.806 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:16.2.14-385.gb14a0a10.el8.x86_64 22/25 2023-12-25T20:49:29.806 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2023-12-25T20:49:29.807 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2023-12-25T20:49:29.807 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /dev 2023-12-25T20:49:29.807 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /run 2023-12-25T20:49:29.807 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2023-12-25T20:49:29.807 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2023-12-25T20:49:29.807 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2023-12-25T20:49:29.807 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2023-12-25T20:49:29.807 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /dev 2023-12-25T20:49:29.807 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:49:30.984 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:16.2.14-385.gb14a0a10.el8.x86_64 23/25 2023-12-25T20:49:31.059 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-385.gb14a0a10.el8.x86_64 23/25 2023-12-25T20:49:31.059 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-25T20:49:31.226 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:49:31.229 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:49:31.229 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2023-12-25T20:49:31.229 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:49:31.229 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-12-25T20:49:31.229 INFO:teuthology.orchestra.run.smithi191.stdout: ceph x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 6.4 k 2023-12-25T20:49:31.229 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2023-12-25T20:49:31.229 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 2.4 M 2023-12-25T20:49:31.229 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 1.6 M 2023-12-25T20:49:31.230 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core noarch 2:16.2.14-385.gb14a0a10.el8 ceph-noarch 245 k 2023-12-25T20:49:31.230 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 5.0 M 2023-12-25T20:49:31.230 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 18 M 2023-12-25T20:49:31.230 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-25T20:49:31.230 INFO:teuthology.orchestra.run.smithi191.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-25T20:49:31.230 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-12-25T20:49:31.230 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-25T20:49:31.230 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-25T20:49:31.230 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-12-25T20:49:31.230 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-25T20:49:31.230 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-25T20:49:31.230 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-25T20:49:31.230 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-12-25T20:49:31.230 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-12-25T20:49:31.230 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-25T20:49:31.230 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-25T20:49:31.231 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-25T20:49:31.231 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-12-25T20:49:31.231 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-12-25T20:49:31.231 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-12-25T20:49:31.231 INFO:teuthology.orchestra.run.smithi191.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-25T20:49:31.231 INFO:teuthology.orchestra.run.smithi191.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-25T20:49:31.231 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-25T20:49:31.231 INFO:teuthology.orchestra.run.smithi191.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-25T20:49:31.231 INFO:teuthology.orchestra.run.smithi191.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-25T20:49:31.231 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-25T20:49:31.231 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-25T20:49:31.231 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-12-25T20:49:31.231 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-25T20:49:31.231 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:49:31.231 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-12-25T20:49:31.232 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:49:31.232 INFO:teuthology.orchestra.run.smithi191.stdout:Install 32 Packages 2023-12-25T20:49:31.232 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:49:31.232 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 31 M 2023-12-25T20:49:31.232 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 101 M 2023-12-25T20:49:31.232 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-12-25T20:49:31.267 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-25T20:49:31.267 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-25T20:49:31.696 INFO:teuthology.orchestra.run.smithi191.stdout:(1/32): ceph-16.2.14-385.gb14a0a10.el8.x86_64.r 48 kB/s | 6.4 kB 00:00 2023-12-25T20:49:31.921 INFO:teuthology.orchestra.run.smithi191.stdout:(2/32): ceph-mgr-16.2.14-385.gb14a0a10.el8.x86_ 4.5 MB/s | 1.6 MB 00:00 2023-12-25T20:49:31.971 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-25T20:49:31.971 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:16.2.14-385.gb14a0a10.el8.x86_64 1/25 2023-12-25T20:49:31.971 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:16.2.14-385.gb14a0a10.el8.x86_64 2/25 2023-12-25T20:49:31.971 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:16.2.14-385.gb14a0a10.el8.x86_64 3/25 2023-12-25T20:49:31.971 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:16.2.14-385.gb14a0a10.el8.x86_64 4/25 2023-12-25T20:49:31.971 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:16.2.14-385.gb14a0a10.el8.x86_64 5/25 2023-12-25T20:49:31.971 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:16.2.14-385.gb14a0a10.el8.x86_6 6/25 2023-12-25T20:49:31.972 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:16.2.14-385.gb14a0a10.el8.x86_64 7/25 2023-12-25T20:49:31.972 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:16.2.14-385.gb14a0a10.el8. 8/25 2023-12-25T20:49:31.972 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:16.2.14-385.gb14a0a10.el8.x8 9/25 2023-12-25T20:49:31.972 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:16.2.14-385.gb14a0a10.el8.x86_64 10/25 2023-12-25T20:49:31.972 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:16.2.14-385.gb14a0a10.el8.x86_64 11/25 2023-12-25T20:49:31.972 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:16.2.14-385.gb14a0a10.el8.x86_64 12/25 2023-12-25T20:49:31.972 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:16.2.14-385.gb14a0a10.el8.x86_64 13/25 2023-12-25T20:49:31.972 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2023-12-25T20:49:31.972 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-12-25T20:49:31.972 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-12-25T20:49:31.972 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-12-25T20:49:31.972 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-12-25T20:49:31.972 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2023-12-25T20:49:31.972 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2023-12-25T20:49:31.972 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2023-12-25T20:49:31.973 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.14-385.gb14a0a10.el8.x86_64 22/25 2023-12-25T20:49:31.973 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-12-25T20:49:31.973 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.14-385.gb14a0a10.el8.x86_64 24/25 2023-12-25T20:49:31.973 INFO:teuthology.orchestra.run.smithi191.stdout:(3/32): ceph-mds-16.2.14-385.gb14a0a10.el8.x86_ 6.0 MB/s | 2.4 MB 00:00 2023-12-25T20:49:32.055 INFO:teuthology.orchestra.run.smithi191.stdout:(4/32): ceph-mgr-modules-core-16.2.14-385.gb14a 2.9 MB/s | 245 kB 00:00 2023-12-25T20:49:32.438 INFO:teuthology.orchestra.run.smithi191.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 409 kB/s | 157 kB 00:00 2023-12-25T20:49:32.539 INFO:teuthology.orchestra.run.smithi191.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 389 kB/s | 39 kB 00:00 2023-12-25T20:49:32.639 INFO:teuthology.orchestra.run.smithi191.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.0 MB/s | 103 kB 00:00 2023-12-25T20:49:32.739 INFO:teuthology.orchestra.run.smithi191.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 570 kB/s | 57 kB 00:00 2023-12-25T20:49:33.081 INFO:teuthology.orchestra.run.smithi191.stdout:(9/32): ceph-osd-16.2.14-385.gb14a0a10.el8.x86_ 16 MB/s | 18 MB 00:01 2023-12-25T20:49:33.106 INFO:teuthology.orchestra.run.smithi191.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.2 MB/s | 457 kB 00:00 2023-12-25T20:49:33.206 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-12-25T20:49:33.206 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:49:33.206 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2023-12-25T20:49:33.206 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:33.206 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:33.207 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-12-25T20:49:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-25T20:49:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: leveldb-1.22-1.el8.x86_64 2023-12-25T20:49:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-25T20:49:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-12-25T20:49:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-1.el8.x86_64 2023-12-25T20:49:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-25T20:49:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-25T20:49:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-25T20:49:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:49:33.208 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:49:33.324 INFO:teuthology.orchestra.run.smithi191.stdout:(11/32): ceph-mon-16.2.14-385.gb14a0a10.el8.x86 3.1 MB/s | 5.0 MB 00:01 2023-12-25T20:49:33.349 INFO:teuthology.orchestra.run.smithi191.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 932 kB/s | 249 kB 00:00 2023-12-25T20:49:33.393 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-test 2023-12-25T20:49:33.415 INFO:teuthology.orchestra.run.smithi191.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 654 kB/s | 44 kB 00:00 2023-12-25T20:49:33.449 INFO:teuthology.orchestra.run.smithi191.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 1.8 MB/s | 641 kB 00:00 2023-12-25T20:49:33.491 INFO:teuthology.orchestra.run.smithi191.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 2.4 MB/s | 185 kB 00:00 2023-12-25T20:49:33.533 INFO:teuthology.orchestra.run.smithi191.stdout:(16/32): python3-pycparser-2.14-14.el8.noarch.r 521 kB/s | 109 kB 00:00 2023-12-25T20:49:33.558 INFO:teuthology.orchestra.run.smithi191.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.6 MB/s | 173 kB 00:00 2023-12-25T20:49:33.592 INFO:teuthology.orchestra.run.smithi191.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 3.7 MB/s | 384 kB 00:00 2023-12-25T20:49:33.617 INFO:teuthology.orchestra.run.smithi191.stdout:(19/32): python3-jaraco-6.2-6.el8.noarch.rpm 133 kB/s | 11 kB 00:00 2023-12-25T20:49:33.642 INFO:teuthology.orchestra.run.smithi191.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 210 kB/s | 18 kB 00:00 2023-12-25T20:49:33.668 INFO:teuthology.orchestra.run.smithi191.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 647 kB/s | 49 kB 00:00 2023-12-25T20:49:33.693 INFO:teuthology.orchestra.run.smithi191.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 778 kB/s | 59 kB 00:00 2023-12-25T20:49:33.718 INFO:teuthology.orchestra.run.smithi191.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 310 kB/s | 16 kB 00:00 2023-12-25T20:49:33.743 INFO:teuthology.orchestra.run.smithi191.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 373 kB/s | 19 kB 00:00 2023-12-25T20:49:33.769 INFO:teuthology.orchestra.run.smithi191.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 478 kB/s | 24 kB 00:00 2023-12-25T20:49:33.794 INFO:teuthology.orchestra.run.smithi191.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 583 kB/s | 29 kB 00:00 2023-12-25T20:49:33.819 INFO:teuthology.orchestra.run.smithi191.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 548 kB/s | 27 kB 00:00 2023-12-25T20:49:33.845 INFO:teuthology.orchestra.run.smithi191.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2023-12-25T20:49:33.870 INFO:teuthology.orchestra.run.smithi191.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2023-12-25T20:49:33.887 INFO:teuthology.orchestra.run.smithi191.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 2.0 MB/s | 85 kB 00:00 2023-12-25T20:49:33.912 INFO:teuthology.orchestra.run.smithi191.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.7 MB/s | 251 kB 00:00 2023-12-25T20:49:33.929 INFO:teuthology.orchestra.run.smithi191.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 391 kB/s | 23 kB 00:00 2023-12-25T20:49:33.930 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:49:33.931 INFO:teuthology.orchestra.run.smithi191.stdout:Total 12 MB/s | 31 MB 00:02 2023-12-25T20:49:34.016 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-12-25T20:49:34.055 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-12-25T20:49:34.055 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-12-25T20:49:34.250 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:48 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:49:34.385 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-12-25T20:49:34.388 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-12-25T20:49:35.497 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-12-25T20:49:35.721 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-12-25T20:49:35.898 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-12-25T20:49:35.927 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:49:35.928 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:49:35.928 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2023-12-25T20:49:35.928 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:49:35.928 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-12-25T20:49:35.929 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 52 M 2023-12-25T20:49:35.929 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2023-12-25T20:49:35.929 INFO:teuthology.orchestra.run.smithi153.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-12-25T20:49:35.929 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 174 k 2023-12-25T20:49:35.929 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-12-25T20:49:35.929 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-12-25T20:49:35.929 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-12-25T20:49:35.929 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:49:35.929 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-12-25T20:49:35.929 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:49:35.929 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2023-12-25T20:49:35.929 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:49:35.930 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 53 M 2023-12-25T20:49:35.930 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 196 M 2023-12-25T20:49:35.930 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-12-25T20:49:36.149 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-12-25T20:49:36.156 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): libcephsqlite-16.2.14-385.gb14a0a10.el8. 942 kB/s | 174 kB 00:00 2023-12-25T20:49:36.404 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-12-25T20:49:36.439 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 434 kB/s | 203 kB 00:00 2023-12-25T20:49:36.540 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 489 kB/s | 187 kB 00:00 2023-12-25T20:49:36.610 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-12-25T20:49:36.615 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.8 MB/s | 323 kB 00:00 2023-12-25T20:49:36.640 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 704 kB/s | 71 kB 00:00 2023-12-25T20:49:36.799 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-12-25T20:49:37.001 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-12-25T20:49:37.203 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-12-25T20:49:37.404 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-12-25T20:49:37.650 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-12-25T20:49:37.914 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-12-25T20:49:38.145 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-12-25T20:49:38.530 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-12-25T20:49:38.615 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): ceph-test-16.2.14-385.gb14a0a10.el8.x86_ 20 MB/s | 52 MB 00:02 2023-12-25T20:49:38.617 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:49:38.618 INFO:teuthology.orchestra.run.smithi153.stdout:Total 20 MB/s | 53 MB 00:02 2023-12-25T20:49:38.660 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-12-25T20:49:38.681 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-12-25T20:49:38.681 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-12-25T20:49:38.784 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-12-25T20:49:38.997 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-12-25T20:49:39.010 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-12-25T20:49:39.013 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-12-25T20:49:39.247 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-12-25T20:49:39.490 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-25T20:49:39.523 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-12-25T20:49:39.732 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-12-25T20:49:39.801 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-12-25T20:49:39.948 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-12-25T20:49:39.978 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-12-25T20:49:40.079 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-25T20:49:40.161 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-25T20:49:40.192 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-12-25T20:49:40.339 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-12-25T20:49:40.427 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-12-25T20:49:40.462 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:16.2.14-385.gb14a0a10.el8.x86_64 5/6 2023-12-25T20:49:40.678 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-12-25T20:49:40.906 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-12-25T20:49:41.165 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-12-25T20:49:41.479 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-12-25T20:49:41.775 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-12-25T20:49:42.149 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-385.gb14a0a10.el8. 27/32 2023-12-25T20:49:42.338 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-2:16.2.14-385.gb14a0a10.el8.x86_64 28/32 2023-12-25T20:49:43.829 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-2:16.2.14-385.gb14a0a10.el8.x86_64 28/32 2023-12-25T20:49:44.014 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-osd-2:16.2.14-385.gb14a0a10.el8.x86_64 29/32 2023-12-25T20:49:44.440 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-osd-2:16.2.14-385.gb14a0a10.el8.x86_64 29/32 2023-12-25T20:49:44.622 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mon-2:16.2.14-385.gb14a0a10.el8.x86_64 30/32 2023-12-25T20:49:44.840 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mon-2:16.2.14-385.gb14a0a10.el8.x86_64 30/32 2023-12-25T20:49:45.005 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mds-2:16.2.14-385.gb14a0a10.el8.x86_64 31/32 2023-12-25T20:49:45.023 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mds-2:16.2.14-385.gb14a0a10.el8.x86_64 31/32 2023-12-25T20:49:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-2:16.2.14-385.gb14a0a10.el8.x86_64 32/32 2023-12-25T20:49:45.149 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:16.2.14-385.gb14a0a10.el8.x86_64 5/6 2023-12-25T20:49:45.402 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:16.2.14-385.gb14a0a10.el8.x86_64 6/6 2023-12-25T20:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-2:16.2.14-385.gb14a0a10.el8.x86_64 32/32 2023-12-25T20:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-2:16.2.14-385.gb14a0a10.el8.x86_64 1/32 2023-12-25T20:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mds-2:16.2.14-385.gb14a0a10.el8.x86_64 2/32 2023-12-25T20:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-2:16.2.14-385.gb14a0a10.el8.x86_64 3/32 2023-12-25T20:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mon-2:16.2.14-385.gb14a0a10.el8.x86_64 4/32 2023-12-25T20:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-osd-2:16.2.14-385.gb14a0a10.el8.x86_64 5/32 2023-12-25T20:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-385.gb14a0a10.el8. 6/32 2023-12-25T20:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-12-25T20:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-12-25T20:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-12-25T20:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-12-25T20:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-12-25T20:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-12-25T20:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-12-25T20:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-12-25T20:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-12-25T20:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-12-25T20:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-25T20:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-12-25T20:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-12-25T20:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-12-25T20:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-12-25T20:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-12-25T20:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-12-25T20:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-12-25T20:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-12-25T20:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-12-25T20:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-12-25T20:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-12-25T20:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-12-25T20:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-12-25T20:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-12-25T20:49:45.691 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-test-2:16.2.14-385.gb14a0a10.el8.x86_64 6/6 2023-12-25T20:49:45.691 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:16.2.14-385.gb14a0a10.el8.x86_64 1/6 2023-12-25T20:49:45.691 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:16.2.14-385.gb14a0a10.el8.x86_64 2/6 2023-12-25T20:49:45.691 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-12-25T20:49:45.692 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-12-25T20:49:45.692 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-12-25T20:49:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-12-25T20:49:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:49:46.425 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-12-25T20:49:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: jq-1.6-8.el8.x86_64 2023-12-25T20:49:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-12-25T20:49:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-1.el8.x86_64 2023-12-25T20:49:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-12-25T20:49:46.426 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:49:46.426 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:49:46.598 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph 2023-12-25T20:49:46.604 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-12-25T20:49:46.604 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:49:46.604 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-12-25T20:49:46.604 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:46.604 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:46.604 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:46.604 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core-2:16.2.14-385.gb14a0a10.el8.noarch 2023-12-25T20:49:46.604 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:46.604 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:49:46.605 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-25T20:49:46.605 INFO:teuthology.orchestra.run.smithi191.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-25T20:49:46.605 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-12-25T20:49:46.605 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-25T20:49:46.605 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-25T20:49:46.605 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-12-25T20:49:46.605 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-25T20:49:46.605 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-25T20:49:46.605 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-25T20:49:46.605 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako-1.0.6-14.el8.noarch 2023-12-25T20:49:46.605 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-25T20:49:46.606 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-25T20:49:46.606 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-25T20:49:46.606 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend-2.6-1.el8.noarch 2023-12-25T20:49:46.606 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-12-25T20:49:46.606 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser-2.14-14.el8.noarch 2023-12-25T20:49:46.606 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz-2017.2-11.el8.noarch 2023-12-25T20:49:46.606 INFO:teuthology.orchestra.run.smithi191.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-25T20:49:46.606 INFO:teuthology.orchestra.run.smithi191.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-25T20:49:46.606 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-25T20:49:46.606 INFO:teuthology.orchestra.run.smithi191.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-25T20:49:46.606 INFO:teuthology.orchestra.run.smithi191.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-25T20:49:46.606 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-25T20:49:46.606 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-25T20:49:46.607 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-25T20:49:46.607 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-25T20:49:46.607 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:49:46.607 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:49:47.018 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-base 2023-12-25T20:49:47.458 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:01 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:49:47.885 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:15 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:49:49.141 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:49:49.145 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:49:49.145 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2023-12-25T20:49:49.145 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:49:49.145 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-12-25T20:49:49.145 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 6.4 k 2023-12-25T20:49:49.145 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2023-12-25T20:49:49.145 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 2.4 M 2023-12-25T20:49:49.145 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 1.6 M 2023-12-25T20:49:49.145 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:16.2.14-385.gb14a0a10.el8 ceph-noarch 245 k 2023-12-25T20:49:49.145 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 5.0 M 2023-12-25T20:49:49.145 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 18 M 2023-12-25T20:49:49.145 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-25T20:49:49.146 INFO:teuthology.orchestra.run.smithi153.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-25T20:49:49.146 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-12-25T20:49:49.146 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-25T20:49:49.146 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-25T20:49:49.146 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-12-25T20:49:49.146 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-25T20:49:49.146 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-25T20:49:49.146 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-25T20:49:49.146 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-12-25T20:49:49.146 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-12-25T20:49:49.146 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-25T20:49:49.146 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-25T20:49:49.147 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-25T20:49:49.147 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-12-25T20:49:49.147 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-12-25T20:49:49.147 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-12-25T20:49:49.147 INFO:teuthology.orchestra.run.smithi153.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-25T20:49:49.147 INFO:teuthology.orchestra.run.smithi153.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-25T20:49:49.147 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-25T20:49:49.147 INFO:teuthology.orchestra.run.smithi153.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-25T20:49:49.147 INFO:teuthology.orchestra.run.smithi153.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-25T20:49:49.147 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-25T20:49:49.147 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-25T20:49:49.147 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-12-25T20:49:49.147 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-25T20:49:49.148 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:49:49.148 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-12-25T20:49:49.148 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:49:49.148 INFO:teuthology.orchestra.run.smithi153.stdout:Install 32 Packages 2023-12-25T20:49:49.148 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:49:49.148 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 31 M 2023-12-25T20:49:49.148 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 101 M 2023-12-25T20:49:49.149 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-12-25T20:49:49.340 INFO:teuthology.orchestra.run.smithi191.stdout:Package ceph-base-2:16.2.14-385.gb14a0a10.el8.x86_64 is already installed. 2023-12-25T20:49:49.566 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:49:49.568 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2023-12-25T20:49:49.568 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:49:49.616 INFO:teuthology.orchestra.run.smithi153.stdout:(1/32): ceph-16.2.14-385.gb14a0a10.el8.x86_64.r 18 kB/s | 6.4 kB 00:00 2023-12-25T20:49:49.675 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install cephadm 2023-12-25T20:49:49.774 INFO:teuthology.orchestra.run.smithi153.stdout:(2/32): ceph-mgr-16.2.14-385.gb14a0a10.el8.x86_ 3.2 MB/s | 1.6 MB 00:00 2023-12-25T20:49:50.308 INFO:teuthology.orchestra.run.smithi153.stdout:(3/32): ceph-mds-16.2.14-385.gb14a0a10.el8.x86_ 2.3 MB/s | 2.4 MB 00:01 2023-12-25T20:49:50.474 INFO:teuthology.orchestra.run.smithi153.stdout:(4/32): ceph-mgr-modules-core-16.2.14-385.gb14a 1.4 MB/s | 245 kB 00:00 2023-12-25T20:49:50.533 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:18 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:49:51.067 INFO:teuthology.orchestra.run.smithi153.stdout:(5/32): ceph-osd-16.2.14-385.gb14a0a10.el8.x86_ 14 MB/s | 18 MB 00:01 2023-12-25T20:49:51.117 INFO:teuthology.orchestra.run.smithi153.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 244 kB/s | 157 kB 00:00 2023-12-25T20:49:51.342 INFO:teuthology.orchestra.run.smithi153.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 460 kB/s | 103 kB 00:00 2023-12-25T20:49:51.467 INFO:teuthology.orchestra.run.smithi153.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 457 kB/s | 57 kB 00:00 2023-12-25T20:49:51.509 INFO:teuthology.orchestra.run.smithi153.stdout:(9/32): python3-markupsafe-0.23-19.el8.x86_64.r 88 kB/s | 39 kB 00:00 2023-12-25T20:49:51.718 INFO:teuthology.orchestra.run.smithi153.stdout:(10/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.2 MB/s | 249 kB 00:00 2023-12-25T20:49:51.993 INFO:teuthology.orchestra.run.smithi153.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 870 kB/s | 457 kB 00:00 2023-12-25T20:49:52.188 INFO:teuthology.orchestra.run.smithi153.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 561 kB/s | 109 kB 00:00 2023-12-25T20:49:52.222 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:49:52.223 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:49:52.223 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2023-12-25T20:49:52.223 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:49:52.223 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-12-25T20:49:52.223 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm noarch 2:16.2.14-385.gb14a0a10.el8 ceph-noarch 85 k 2023-12-25T20:49:52.223 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:49:52.224 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-12-25T20:49:52.224 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:49:52.224 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2023-12-25T20:49:52.224 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:49:52.224 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 85 k 2023-12-25T20:49:52.224 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 352 k 2023-12-25T20:49:52.224 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-12-25T20:49:52.327 INFO:teuthology.orchestra.run.smithi153.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 314 kB/s | 44 kB 00:00 2023-12-25T20:49:52.402 INFO:teuthology.orchestra.run.smithi153.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 2.4 MB/s | 185 kB 00:00 2023-12-25T20:49:52.478 INFO:teuthology.orchestra.run.smithi153.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.2 MB/s | 173 kB 00:00 2023-12-25T20:49:52.691 INFO:teuthology.orchestra.run.smithi191.stdout:cephadm-16.2.14-385.gb14a0a10.el8.noarch.rpm 183 kB/s | 85 kB 00:00 2023-12-25T20:49:52.692 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:49:52.692 INFO:teuthology.orchestra.run.smithi191.stdout:Total 182 kB/s | 85 kB 00:00 2023-12-25T20:49:52.692 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-12-25T20:49:52.699 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-12-25T20:49:52.699 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-12-25T20:49:52.701 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-12-25T20:49:52.704 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-12-25T20:49:52.728 INFO:teuthology.orchestra.run.smithi153.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.5 MB/s | 384 kB 00:00 2023-12-25T20:49:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-12-25T20:49:52.770 INFO:teuthology.orchestra.run.smithi153.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 267 kB/s | 11 kB 00:00 2023-12-25T20:49:52.812 INFO:teuthology.orchestra.run.smithi153.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 421 kB/s | 18 kB 00:00 2023-12-25T20:49:52.862 INFO:teuthology.orchestra.run.smithi153.stdout:(19/32): python3-logutils-0.3.5-11.el8.noarch.r 977 kB/s | 49 kB 00:00 2023-12-25T20:49:52.921 INFO:teuthology.orchestra.run.smithi153.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 1.0 MB/s | 59 kB 00:00 2023-12-25T20:49:53.079 INFO:teuthology.orchestra.run.smithi153.stdout:(21/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.7 MB/s | 283 kB 00:00 2023-12-25T20:49:53.122 INFO:teuthology.orchestra.run.smithi153.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 374 kB/s | 16 kB 00:00 2023-12-25T20:49:53.189 INFO:teuthology.orchestra.run.smithi153.stdout:(23/32): ceph-mon-16.2.14-385.gb14a0a10.el8.x86 1.4 MB/s | 5.0 MB 00:03 2023-12-25T20:49:53.214 INFO:teuthology.orchestra.run.smithi153.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 204 kB/s | 19 kB 00:00 2023-12-25T20:49:53.264 INFO:teuthology.orchestra.run.smithi153.stdout:(25/32): python3-tempora-1.14.1-5.el8.noarch.rp 647 kB/s | 29 kB 00:00 2023-12-25T20:49:53.282 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:16.2.14-385.gb14a0a10.el8.noarch 1/1 2023-12-25T20:49:53.306 INFO:teuthology.orchestra.run.smithi153.stdout:(26/32): python3-trustme-0.6.0-4.el8.noarch.rpm 668 kB/s | 27 kB 00:00 2023-12-25T20:49:53.364 INFO:teuthology.orchestra.run.smithi153.stdout:(27/32): python3-singledispatch-3.4.0.3-18.el8. 136 kB/s | 24 kB 00:00 2023-12-25T20:49:53.398 INFO:teuthology.orchestra.run.smithi153.stdout:(28/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.7 MB/s | 255 kB 00:00 2023-12-25T20:49:53.432 INFO:teuthology.orchestra.run.smithi153.stdout:(29/32): python3-cryptography-3.2.1-6.el8.x86_6 374 kB/s | 641 kB 00:01 2023-12-25T20:49:53.440 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephadm-2:16.2.14-385.gb14a0a10.el8.noarch 1/1 2023-12-25T20:49:53.457 INFO:teuthology.orchestra.run.smithi153.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.4 MB/s | 85 kB 00:00 2023-12-25T20:49:53.515 INFO:teuthology.orchestra.run.smithi153.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.6 MB/s | 251 kB 00:00 2023-12-25T20:49:53.632 INFO:teuthology.orchestra.run.smithi153.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 115 kB/s | 23 kB 00:00 2023-12-25T20:49:53.636 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:49:53.636 INFO:teuthology.orchestra.run.smithi153.stdout:Total 7.0 MB/s | 31 MB 00:04 2023-12-25T20:49:53.663 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:16.2.14-385.gb14a0a10.el8.noarch 1/1 2023-12-25T20:49:53.720 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-12-25T20:49:53.759 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-12-25T20:49:53.759 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-12-25T20:49:54.080 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-12-25T20:49:54.084 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-12-25T20:49:54.334 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephadm-2:16.2.14-385.gb14a0a10.el8.noarch 1/1 2023-12-25T20:49:54.334 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:49:54.334 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-12-25T20:49:54.335 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm-2:16.2.14-385.gb14a0a10.el8.noarch 2023-12-25T20:49:54.335 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:49:54.335 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:49:54.573 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-immutable-object-cache 2023-12-25T20:49:55.209 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-12-25T20:49:55.415 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-12-25T20:49:55.428 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:23 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:49:55.618 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-12-25T20:49:55.817 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-12-25T20:49:56.033 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-12-25T20:49:56.214 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-12-25T20:49:56.394 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-12-25T20:49:56.554 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-12-25T20:49:56.740 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-12-25T20:49:56.956 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-12-25T20:49:57.138 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:49:57.139 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:49:57.139 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2023-12-25T20:49:57.140 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:49:57.140 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-12-25T20:49:57.140 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 161 k 2023-12-25T20:49:57.140 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:49:57.140 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-12-25T20:49:57.140 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:49:57.140 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2023-12-25T20:49:57.140 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:49:57.140 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 161 k 2023-12-25T20:49:57.140 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 470 k 2023-12-25T20:49:57.140 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-12-25T20:49:57.194 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-12-25T20:49:57.407 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-12-25T20:49:57.623 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-12-25T20:49:57.770 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-immutable-object-cache-16.2.14-385.gb14a0a 256 kB/s | 161 kB 00:00 2023-12-25T20:49:57.771 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:49:57.771 INFO:teuthology.orchestra.run.smithi191.stdout:Total 255 kB/s | 161 kB 00:00 2023-12-25T20:49:57.771 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-12-25T20:49:57.783 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-12-25T20:49:57.784 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-12-25T20:49:57.837 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-12-25T20:49:57.840 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-12-25T20:49:58.004 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-12-25T20:49:58.254 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-12-25T20:49:58.442 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-12-25T20:49:58.667 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-12-25T20:49:58.675 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-12-25T20:49:58.869 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-25T20:49:59.019 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-385.gb14a0a10. 1/1 2023-12-25T20:49:59.214 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-12-25T20:49:59.389 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-12-25T20:49:59.447 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-385.gb14a0a10. 1/1 2023-12-25T20:49:59.637 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-12-25T20:49:59.806 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-12-25T20:50:00.109 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-385.gb14a0a10. 1/1 2023-12-25T20:50:00.110 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:00.110 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-12-25T20:50:00.110 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:50:00.110 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:00.110 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:50:00.221 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-12-25T20:50:00.312 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr 2023-12-25T20:50:00.500 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-12-25T20:50:00.760 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-12-25T20:50:01.066 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-12-25T20:50:01.170 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:29 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:50:01.353 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-12-25T20:50:01.690 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-385.gb14a0a10.el8. 27/32 2023-12-25T20:50:01.879 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:16.2.14-385.gb14a0a10.el8.x86_64 28/32 2023-12-25T20:50:02.628 INFO:teuthology.orchestra.run.smithi191.stdout:Package ceph-mgr-2:16.2.14-385.gb14a0a10.el8.x86_64 is already installed. 2023-12-25T20:50:02.858 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:50:02.859 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2023-12-25T20:50:02.859 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:50:02.968 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-dashboard 2023-12-25T20:50:03.381 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:16.2.14-385.gb14a0a10.el8.x86_64 28/32 2023-12-25T20:50:03.551 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:16.2.14-385.gb14a0a10.el8.x86_64 29/32 2023-12-25T20:50:03.825 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:31 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:50:03.981 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:16.2.14-385.gb14a0a10.el8.x86_64 29/32 2023-12-25T20:50:04.143 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:16.2.14-385.gb14a0a10.el8.x86_64 30/32 2023-12-25T20:50:04.360 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:16.2.14-385.gb14a0a10.el8.x86_64 30/32 2023-12-25T20:50:04.550 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:16.2.14-385.gb14a0a10.el8.x86_64 31/32 2023-12-25T20:50:04.568 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:16.2.14-385.gb14a0a10.el8.x86_64 31/32 2023-12-25T20:50:04.675 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:16.2.14-385.gb14a0a10.el8.x86_64 32/32 2023-12-25T20:50:05.081 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-2:16.2.14-385.gb14a0a10.el8.x86_64 32/32 2023-12-25T20:50:05.081 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:16.2.14-385.gb14a0a10.el8.x86_64 1/32 2023-12-25T20:50:05.082 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:16.2.14-385.gb14a0a10.el8.x86_64 2/32 2023-12-25T20:50:05.082 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:16.2.14-385.gb14a0a10.el8.x86_64 3/32 2023-12-25T20:50:05.082 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:16.2.14-385.gb14a0a10.el8.x86_64 4/32 2023-12-25T20:50:05.082 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:16.2.14-385.gb14a0a10.el8.x86_64 5/32 2023-12-25T20:50:05.082 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-385.gb14a0a10.el8. 6/32 2023-12-25T20:50:05.082 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-12-25T20:50:05.082 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-12-25T20:50:05.082 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-12-25T20:50:05.082 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-12-25T20:50:05.082 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-12-25T20:50:05.082 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-12-25T20:50:05.082 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-12-25T20:50:05.082 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-12-25T20:50:05.082 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-12-25T20:50:05.083 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-12-25T20:50:05.083 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-25T20:50:05.083 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-12-25T20:50:05.083 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-12-25T20:50:05.083 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-12-25T20:50:05.083 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-12-25T20:50:05.083 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-12-25T20:50:05.083 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-12-25T20:50:05.083 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-12-25T20:50:05.083 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-12-25T20:50:05.083 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-12-25T20:50:05.083 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-12-25T20:50:05.083 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-12-25T20:50:05.083 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-12-25T20:50:05.084 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-12-25T20:50:05.084 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-12-25T20:50:05.519 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:50:05.520 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:50:05.520 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2023-12-25T20:50:05.520 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:50:05.520 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-12-25T20:50:05.521 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard noarch 2:16.2.14-385.gb14a0a10.el8 ceph-noarch 3.1 M 2023-12-25T20:50:05.521 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2023-12-25T20:50:05.521 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards noarch 2:16.2.14-385.gb14a0a10.el8 ceph-noarch 24 k 2023-12-25T20:50:05.521 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts noarch 2:16.2.14-385.gb14a0a10.el8 ceph-noarch 15 k 2023-12-25T20:50:05.521 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-12-25T20:50:05.521 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-25T20:50:05.521 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-25T20:50:05.521 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:05.521 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-12-25T20:50:05.521 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:50:05.521 INFO:teuthology.orchestra.run.smithi191.stdout:Install 6 Packages 2023-12-25T20:50:05.521 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:05.522 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 3.4 M 2023-12-25T20:50:05.522 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 22 M 2023-12-25T20:50:05.522 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-12-25T20:50:06.455 INFO:teuthology.orchestra.run.smithi191.stdout:(1/6): ceph-prometheus-alerts-16.2.14-385.gb14a 101 kB/s | 15 kB 00:00 2023-12-25T20:50:06.522 INFO:teuthology.orchestra.run.smithi191.stdout:(2/6): ceph-grafana-dashboards-16.2.14-385.gb14 114 kB/s | 24 kB 00:00 2023-12-25T20:50:06.573 INFO:teuthology.orchestra.run.smithi191.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 675 kB/s | 34 kB 00:00 2023-12-25T20:50:06.589 INFO:teuthology.orchestra.run.smithi191.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 318 kB/s | 43 kB 00:00 2023-12-25T20:50:06.631 INFO:teuthology.orchestra.run.smithi191.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 3.3 MB/s | 196 kB 00:00 2023-12-25T20:50:06.678 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-12-25T20:50:06.678 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:06.678 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-12-25T20:50:06.678 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:50:06.678 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:50:06.678 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:50:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:16.2.14-385.gb14a0a10.el8.noarch 2023-12-25T20:50:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:50:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:50:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-25T20:50:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-25T20:50:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-12-25T20:50:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-25T20:50:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-25T20:50:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-12-25T20:50:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-25T20:50:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-25T20:50:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-25T20:50:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.0.6-14.el8.noarch 2023-12-25T20:50:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-25T20:50:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-25T20:50:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-25T20:50:06.680 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-2.6-1.el8.noarch 2023-12-25T20:50:06.680 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-12-25T20:50:06.680 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.14-14.el8.noarch 2023-12-25T20:50:06.680 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2017.2-11.el8.noarch 2023-12-25T20:50:06.680 INFO:teuthology.orchestra.run.smithi153.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-25T20:50:06.680 INFO:teuthology.orchestra.run.smithi153.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-25T20:50:06.680 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-25T20:50:06.680 INFO:teuthology.orchestra.run.smithi153.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-25T20:50:06.680 INFO:teuthology.orchestra.run.smithi153.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-25T20:50:06.680 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-25T20:50:06.680 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-25T20:50:06.680 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-25T20:50:06.680 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-25T20:50:06.680 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:06.681 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:50:06.935 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-base 2023-12-25T20:50:07.048 INFO:teuthology.orchestra.run.smithi191.stdout:(6/6): ceph-mgr-dashboard-16.2.14-385.gb14a0a10 4.2 MB/s | 3.1 MB 00:00 2023-12-25T20:50:07.050 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:50:07.051 INFO:teuthology.orchestra.run.smithi191.stdout:Total 2.2 MB/s | 3.4 MB 00:01 2023-12-25T20:50:07.062 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-12-25T20:50:07.073 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-12-25T20:50:07.073 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-12-25T20:50:07.270 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-12-25T20:50:07.273 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-12-25T20:50:07.546 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-12-25T20:50:07.731 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-12-25T20:50:07.794 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:21 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:50:07.935 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-12-25T20:50:08.128 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-12-25T20:50:08.240 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-385.gb14a0a10.el8.n 4/6 2023-12-25T20:50:09.237 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-385.gb14a0a10.el8. 5/6 2023-12-25T20:50:09.255 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-base-2:16.2.14-385.gb14a0a10.el8.x86_64 is already installed. 2023-12-25T20:50:09.479 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:50:09.480 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2023-12-25T20:50:09.480 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:50:09.571 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install cephadm 2023-12-25T20:50:09.949 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-385.gb14a0a10.el8.noarc 6/6 2023-12-25T20:50:10.347 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-385.gb14a0a10.el8.noarc 6/6 2023-12-25T20:50:10.348 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-385.gb14a0a10.el8. 1/6 2023-12-25T20:50:10.348 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-385.gb14a0a10.el8.noarc 2/6 2023-12-25T20:50:10.348 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-385.gb14a0a10.el8.n 3/6 2023-12-25T20:50:10.348 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-12-25T20:50:10.348 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-12-25T20:50:10.431 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:24 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:50:11.023 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-12-25T20:50:11.023 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:11.023 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-12-25T20:50:11.023 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards-2:16.2.14-385.gb14a0a10.el8.noarch 2023-12-25T20:50:11.023 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard-2:16.2.14-385.gb14a0a10.el8.noarch 2023-12-25T20:50:11.023 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts-2:16.2.14-385.gb14a0a10.el8.noarch 2023-12-25T20:50:11.023 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-25T20:50:11.023 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-25T20:50:11.023 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-25T20:50:11.023 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:11.023 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:50:11.141 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-25T20:50:11.998 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:39 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:50:12.122 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:50:12.123 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:50:12.123 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2023-12-25T20:50:12.123 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:50:12.123 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-12-25T20:50:12.123 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:16.2.14-385.gb14a0a10.el8 ceph-noarch 85 k 2023-12-25T20:50:12.123 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-12-25T20:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2023-12-25T20:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 85 k 2023-12-25T20:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 352 k 2023-12-25T20:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-12-25T20:50:12.687 INFO:teuthology.orchestra.run.smithi153.stdout:cephadm-16.2.14-385.gb14a0a10.el8.noarch.rpm 152 kB/s | 85 kB 00:00 2023-12-25T20:50:12.687 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:50:12.687 INFO:teuthology.orchestra.run.smithi153.stdout:Total 151 kB/s | 85 kB 00:00 2023-12-25T20:50:12.688 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-12-25T20:50:12.695 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-12-25T20:50:12.695 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-12-25T20:50:12.697 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-12-25T20:50:12.700 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-12-25T20:50:12.749 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-12-25T20:50:13.214 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:16.2.14-385.gb14a0a10.el8.noarch 1/1 2023-12-25T20:50:13.399 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:16.2.14-385.gb14a0a10.el8.noarch 1/1 2023-12-25T20:50:13.632 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:16.2.14-385.gb14a0a10.el8.noarch 1/1 2023-12-25T20:50:13.683 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:50:13.684 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2023-12-25T20:50:13.684 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2023-12-25T20:50:13.684 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2023-12-25T20:50:13.684 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-12-25T20:50:13.684 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-385.gb14a0a10.el8 ceph-noarch 8.6 M 2023-12-25T20:50:13.684 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2023-12-25T20:50:13.684 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-12-25T20:50:13.684 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-12-25T20:50:13.685 INFO:teuthology.orchestra.run.smithi191.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-12-25T20:50:13.685 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-12-25T20:50:13.685 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-12-25T20:50:13.685 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-12-25T20:50:13.685 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2023-12-25T20:50:13.685 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:13.685 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-12-25T20:50:13.685 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2023-12-25T20:50:13.685 INFO:teuthology.orchestra.run.smithi191.stdout:Install 8 Packages 2023-12-25T20:50:13.685 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:13.686 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 37 M 2023-12-25T20:50:13.686 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 210 M 2023-12-25T20:50:13.686 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-12-25T20:50:14.303 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:16.2.14-385.gb14a0a10.el8.noarch 1/1 2023-12-25T20:50:14.304 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:14.304 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-12-25T20:50:14.304 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:16.2.14-385.gb14a0a10.el8.noarch 2023-12-25T20:50:14.304 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:14.304 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:50:14.522 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-immutable-object-cache 2023-12-25T20:50:14.702 INFO:teuthology.orchestra.run.smithi191.stdout:(1/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 6.2 MB/s | 4.9 MB 00:00 2023-12-25T20:50:14.786 INFO:teuthology.orchestra.run.smithi191.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 5.4 MB/s | 4.7 MB 00:00 2023-12-25T20:50:14.903 INFO:teuthology.orchestra.run.smithi191.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.9 MB/s | 225 kB 00:00 2023-12-25T20:50:15.011 INFO:teuthology.orchestra.run.smithi191.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 12 MB/s | 3.7 MB 00:00 2023-12-25T20:50:15.170 INFO:teuthology.orchestra.run.smithi191.stdout:(5/8): ceph-mgr-diskprediction-local-16.2.14-38 6.9 MB/s | 8.6 MB 00:01 2023-12-25T20:50:15.345 INFO:teuthology.orchestra.run.smithi191.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 987 kB/s | 172 kB 00:00 2023-12-25T20:50:15.379 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:29 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:50:16.440 INFO:teuthology.orchestra.run.smithi191.stdout:(7/8): python3-scipy-1.0.0-21.module_el8+762+77 9.2 MB/s | 14 MB 00:01 2023-12-25T20:50:17.047 INFO:teuthology.orchestra.run.smithi191.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 317 kB/s | 645 kB 00:02 2023-12-25T20:50:17.048 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:50:17.048 INFO:teuthology.orchestra.run.smithi191.stdout:Total 11 MB/s | 37 MB 00:03 2023-12-25T20:50:17.080 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:50:17.081 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:50:17.081 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2023-12-25T20:50:17.081 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:50:17.081 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-12-25T20:50:17.081 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 161 k 2023-12-25T20:50:17.081 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:17.081 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-12-25T20:50:17.081 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:50:17.081 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2023-12-25T20:50:17.081 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:17.082 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 161 k 2023-12-25T20:50:17.082 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 470 k 2023-12-25T20:50:17.082 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-12-25T20:50:17.284 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-12-25T20:50:17.302 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-12-25T20:50:17.302 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-12-25T20:50:17.628 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-12-25T20:50:17.631 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-12-25T20:50:17.992 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-immutable-object-cache-16.2.14-385.gb14a0a 177 kB/s | 161 kB 00:00 2023-12-25T20:50:17.993 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:50:17.994 INFO:teuthology.orchestra.run.smithi153.stdout:Total 177 kB/s | 161 kB 00:00 2023-12-25T20:50:17.994 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-12-25T20:50:18.006 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-12-25T20:50:18.006 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-12-25T20:50:18.057 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-12-25T20:50:18.060 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-12-25T20:50:18.174 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-12-25T20:50:18.188 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-12-25T20:50:18.357 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-25T20:50:18.373 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-385.gb14a0a10. 1/1 2023-12-25T20:50:18.478 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-25T20:50:18.632 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-25T20:50:18.795 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-385.gb14a0a10. 1/1 2023-12-25T20:50:19.209 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-25T20:50:19.399 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-12-25T20:50:19.489 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-385.gb14a0a10. 1/1 2023-12-25T20:50:19.490 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:19.490 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-12-25T20:50:19.490 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:50:19.490 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:19.490 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:50:19.714 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr 2023-12-25T20:50:20.016 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-12-25T20:50:20.193 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-25T20:50:20.575 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:34 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:50:20.876 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-25T20:50:21.209 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-12-25T20:50:21.463 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-25T20:50:22.053 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-mgr-2:16.2.14-385.gb14a0a10.el8.x86_64 is already installed. 2023-12-25T20:50:22.283 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:50:22.284 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2023-12-25T20:50:22.284 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:50:22.380 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-dashboard 2023-12-25T20:50:23.231 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-25T20:50:23.234 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:37 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:50:24.530 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2023-12-25T20:50:24.714 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-385.gb14a0a1 8/8 2023-12-25T20:50:24.940 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:50:24.941 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:50:24.941 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2023-12-25T20:50:24.942 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:50:24.942 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-12-25T20:50:24.942 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:16.2.14-385.gb14a0a10.el8 ceph-noarch 3.1 M 2023-12-25T20:50:24.942 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2023-12-25T20:50:24.942 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:16.2.14-385.gb14a0a10.el8 ceph-noarch 24 k 2023-12-25T20:50:24.942 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:16.2.14-385.gb14a0a10.el8 ceph-noarch 15 k 2023-12-25T20:50:24.942 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-12-25T20:50:24.942 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-25T20:50:24.942 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-25T20:50:24.942 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:24.942 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-12-25T20:50:24.942 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:50:24.942 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2023-12-25T20:50:24.942 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:24.943 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 3.4 M 2023-12-25T20:50:24.943 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 22 M 2023-12-25T20:50:24.943 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-12-25T20:50:25.065 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-385.gb14a0a1 8/8 2023-12-25T20:50:25.065 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-385.gb14a0a1 1/8 2023-12-25T20:50:25.066 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-12-25T20:50:25.066 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-12-25T20:50:25.066 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-12-25T20:50:25.066 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-12-25T20:50:25.066 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2023-12-25T20:50:25.066 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-12-25T20:50:25.421 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): ceph-prometheus-alerts-16.2.14-385.gb14a 67 kB/s | 15 kB 00:00 2023-12-25T20:50:25.479 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): ceph-grafana-dashboards-16.2.14-385.gb14 87 kB/s | 24 kB 00:00 2023-12-25T20:50:25.596 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 243 kB/s | 43 kB 00:00 2023-12-25T20:50:25.622 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 237 kB/s | 34 kB 00:00 2023-12-25T20:50:25.770 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-12-25T20:50:25.770 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:25.770 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-12-25T20:50:25.770 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local-2:16.2.14-385.gb14a0a10.el8.noarch 2023-12-25T20:50:25.770 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-12-25T20:50:25.770 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-12-25T20:50:25.770 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-0.3.15-4.el8.x86_64 2023-12-25T20:50:25.770 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-12-25T20:50:25.771 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-12-25T20:50:25.771 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-12-25T20:50:25.771 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2023-12-25T20:50:25.771 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:25.771 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:50:25.780 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.0 MB/s | 196 kB 00:00 2023-12-25T20:50:25.831 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): ceph-mgr-dashboard-16.2.14-385.gb14a0a10 5.0 MB/s | 3.1 MB 00:00 2023-12-25T20:50:25.834 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:50:25.834 INFO:teuthology.orchestra.run.smithi153.stdout:Total 3.9 MB/s | 3.4 MB 00:00 2023-12-25T20:50:25.845 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-12-25T20:50:25.853 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-12-25T20:50:25.854 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-12-25T20:50:25.887 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-rook 2023-12-25T20:50:25.989 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-12-25T20:50:25.992 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-12-25T20:50:26.284 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-12-25T20:50:26.444 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-12-25T20:50:26.656 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-12-25T20:50:26.742 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:54 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:50:26.882 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-12-25T20:50:27.011 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-385.gb14a0a10.el8.n 4/6 2023-12-25T20:50:28.027 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-385.gb14a0a10.el8. 5/6 2023-12-25T20:50:28.308 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-385.gb14a0a10.el8.noarc 6/6 2023-12-25T20:50:28.426 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:50:28.427 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:50:28.427 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2023-12-25T20:50:28.428 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:50:28.428 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-12-25T20:50:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook noarch 2:16.2.14-385.gb14a0a10.el8 ceph-noarch 27 k 2023-12-25T20:50:28.428 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2023-12-25T20:50:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-25T20:50:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-25T20:50:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-25T20:50:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-12-25T20:50:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-12-25T20:50:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-25T20:50:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-12-25T20:50:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-12-25T20:50:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-12-25T20:50:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-12-25T20:50:28.429 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-25T20:50:28.429 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-25T20:50:28.429 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:28.429 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-12-25T20:50:28.429 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:50:28.429 INFO:teuthology.orchestra.run.smithi191.stdout:Install 13 Packages 2023-12-25T20:50:28.429 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:28.429 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 1.9 M 2023-12-25T20:50:28.430 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 24 M 2023-12-25T20:50:28.430 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-12-25T20:50:28.594 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-385.gb14a0a10.el8.noarc 6/6 2023-12-25T20:50:28.595 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-385.gb14a0a10.el8. 1/6 2023-12-25T20:50:28.595 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-385.gb14a0a10.el8.noarc 2/6 2023-12-25T20:50:28.595 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-385.gb14a0a10.el8.n 3/6 2023-12-25T20:50:28.595 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-12-25T20:50:28.595 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-12-25T20:50:29.391 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-12-25T20:50:29.392 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:29.392 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-12-25T20:50:29.392 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:16.2.14-385.gb14a0a10.el8.noarch 2023-12-25T20:50:29.392 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:16.2.14-385.gb14a0a10.el8.noarch 2023-12-25T20:50:29.392 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:16.2.14-385.gb14a0a10.el8.noarch 2023-12-25T20:50:29.393 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-25T20:50:29.393 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-25T20:50:29.393 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-25T20:50:29.393 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:29.393 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:50:29.535 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-25T20:50:30.393 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:44 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:50:30.735 INFO:teuthology.orchestra.run.smithi191.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 109 kB/s | 27 kB 00:00 2023-12-25T20:50:30.752 INFO:teuthology.orchestra.run.smithi191.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 77 kB/s | 20 kB 00:00 2023-12-25T20:50:30.919 INFO:teuthology.orchestra.run.smithi191.stdout:(3/13): ceph-mgr-rook-16.2.14-385.gb14a0a10.el8 64 kB/s | 27 kB 00:00 2023-12-25T20:50:30.969 INFO:teuthology.orchestra.run.smithi191.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 541 kB/s | 126 kB 00:00 2023-12-25T20:50:30.995 INFO:teuthology.orchestra.run.smithi191.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 454 kB/s | 110 kB 00:00 2023-12-25T20:50:31.095 INFO:teuthology.orchestra.run.smithi191.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 340 kB/s | 43 kB 00:00 2023-12-25T20:50:31.112 INFO:teuthology.orchestra.run.smithi191.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 805 kB/s | 155 kB 00:00 2023-12-25T20:50:31.171 INFO:teuthology.orchestra.run.smithi191.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 188 kB/s | 33 kB 00:00 2023-12-25T20:50:31.238 INFO:teuthology.orchestra.run.smithi191.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 114 kB/s | 16 kB 00:00 2023-12-25T20:50:31.305 INFO:teuthology.orchestra.run.smithi191.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 422 kB/s | 81 kB 00:00 2023-12-25T20:50:31.363 INFO:teuthology.orchestra.run.smithi191.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 513 kB/s | 64 kB 00:00 2023-12-25T20:50:31.380 INFO:teuthology.orchestra.run.smithi191.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 814 kB/s | 61 kB 00:00 2023-12-25T20:50:31.481 INFO:teuthology.orchestra.run.smithi191.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.8 MB/s | 1.2 MB 00:00 2023-12-25T20:50:31.481 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:50:31.481 INFO:teuthology.orchestra.run.smithi191.stdout:Total 641 kB/s | 1.9 MB 00:03 2023-12-25T20:50:31.539 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-12-25T20:50:31.554 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-12-25T20:50:31.554 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-12-25T20:50:31.641 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-12-25T20:50:31.644 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-12-25T20:50:32.089 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:50:32.090 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2023-12-25T20:50:32.090 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2023-12-25T20:50:32.090 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2023-12-25T20:50:32.090 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-12-25T20:50:32.090 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-385.gb14a0a10.el8 ceph-noarch 8.6 M 2023-12-25T20:50:32.090 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2023-12-25T20:50:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-12-25T20:50:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-12-25T20:50:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-12-25T20:50:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-12-25T20:50:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-12-25T20:50:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-12-25T20:50:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2023-12-25T20:50:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:32.091 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-12-25T20:50:32.091 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2023-12-25T20:50:32.091 INFO:teuthology.orchestra.run.smithi153.stdout:Install 8 Packages 2023-12-25T20:50:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:32.092 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 37 M 2023-12-25T20:50:32.092 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 210 M 2023-12-25T20:50:32.092 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-12-25T20:50:32.153 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-12-25T20:50:32.433 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-25T20:50:32.654 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-12-25T20:50:32.857 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-12-25T20:50:32.985 INFO:teuthology.orchestra.run.smithi153.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 5.8 MB/s | 4.7 MB 00:00 2023-12-25T20:50:33.052 INFO:teuthology.orchestra.run.smithi153.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.5 MB/s | 4.9 MB 00:00 2023-12-25T20:50:33.069 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-12-25T20:50:33.152 INFO:teuthology.orchestra.run.smithi153.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.2 MB/s | 225 kB 00:00 2023-12-25T20:50:33.293 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-12-25T20:50:33.327 INFO:teuthology.orchestra.run.smithi153.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 11 MB/s | 3.7 MB 00:00 2023-12-25T20:50:33.491 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-12-25T20:50:33.776 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-12-25T20:50:34.042 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-12-25T20:50:34.404 INFO:teuthology.orchestra.run.smithi153.stdout:(5/8): ceph-mgr-diskprediction-local-16.2.14-38 3.8 MB/s | 8.6 MB 00:02 2023-12-25T20:50:34.608 INFO:teuthology.orchestra.run.smithi153.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 504 kB/s | 645 kB 00:01 2023-12-25T20:50:34.638 INFO:teuthology.orchestra.run.smithi153.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 739 kB/s | 172 kB 00:00 2023-12-25T20:50:34.780 INFO:teuthology.orchestra.run.smithi153.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 8.7 MB/s | 14 MB 00:01 2023-12-25T20:50:34.781 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:50:34.781 INFO:teuthology.orchestra.run.smithi153.stdout:Total 14 MB/s | 37 MB 00:02 2023-12-25T20:50:35.018 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-12-25T20:50:35.036 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-12-25T20:50:35.036 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-12-25T20:50:35.074 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-12-25T20:50:35.344 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-12-25T20:50:35.351 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-12-25T20:50:35.355 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-12-25T20:50:35.528 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-12-25T20:50:35.745 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-12-25T20:50:35.904 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-rook-2:16.2.14-385.gb14a0a10.el8.noarch 13/13 2023-12-25T20:50:35.904 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-12-25T20:50:36.062 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-25T20:50:36.189 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-385.gb14a0a10.el8.noarch 13/13 2023-12-25T20:50:36.189 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-rook-2:16.2.14-385.gb14a0a10.el8.noarch 1/13 2023-12-25T20:50:36.190 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-12-25T20:50:36.190 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-12-25T20:50:36.190 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-12-25T20:50:36.190 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-12-25T20:50:36.190 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-12-25T20:50:36.190 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-12-25T20:50:36.190 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-12-25T20:50:36.190 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-25T20:50:36.190 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-12-25T20:50:36.190 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-12-25T20:50:36.190 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-12-25T20:50:36.195 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-25T20:50:36.353 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-25T20:50:36.949 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-12-25T20:50:36.949 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:36.949 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-12-25T20:50:36.949 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook-2:16.2.14-385.gb14a0a10.el8.noarch 2023-12-25T20:50:36.949 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-25T20:50:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-25T20:50:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-25T20:50:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-25T20:50:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-25T20:50:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-25T20:50:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-25T20:50:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-25T20:50:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-25T20:50:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-25T20:50:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-25T20:50:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-25T20:50:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:36.950 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:50:36.953 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-25T20:50:37.071 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-cephadm 2023-12-25T20:50:37.136 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-12-25T20:50:37.769 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-12-25T20:50:37.927 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:05 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:50:37.967 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-25T20:50:38.640 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-25T20:50:38.956 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-12-25T20:50:39.194 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-25T20:50:39.630 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:50:39.631 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:50:39.631 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2023-12-25T20:50:39.631 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:50:39.631 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-12-25T20:50:39.632 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm noarch 2:16.2.14-385.gb14a0a10.el8 ceph-noarch 116 k 2023-12-25T20:50:39.632 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2023-12-25T20:50:39.632 INFO:teuthology.orchestra.run.smithi191.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-25T20:50:39.632 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-12-25T20:50:39.632 INFO:teuthology.orchestra.run.smithi191.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-25T20:50:39.632 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-12-25T20:50:39.632 INFO:teuthology.orchestra.run.smithi191.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-25T20:50:39.632 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:39.632 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-12-25T20:50:39.632 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:50:39.632 INFO:teuthology.orchestra.run.smithi191.stdout:Install 6 Packages 2023-12-25T20:50:39.633 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:39.633 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 5.8 M 2023-12-25T20:50:39.633 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 24 M 2023-12-25T20:50:39.633 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-12-25T20:50:40.220 INFO:teuthology.orchestra.run.smithi191.stdout:(1/6): ceph-mgr-cephadm-16.2.14-385.gb14a0a10.e 247 kB/s | 116 kB 00:00 2023-12-25T20:50:40.287 INFO:teuthology.orchestra.run.smithi191.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.0 MB/s | 538 kB 00:00 2023-12-25T20:50:40.304 INFO:teuthology.orchestra.run.smithi191.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 254 kB/s | 21 kB 00:00 2023-12-25T20:50:40.363 INFO:teuthology.orchestra.run.smithi191.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 837 kB/s | 49 kB 00:00 2023-12-25T20:50:40.467 INFO:teuthology.orchestra.run.smithi191.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.0 MB/s | 355 kB 00:00 2023-12-25T20:50:40.697 INFO:teuthology.orchestra.run.smithi191.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 5.1 MB/s | 4.8 MB 00:00 2023-12-25T20:50:40.697 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:50:40.697 INFO:teuthology.orchestra.run.smithi191.stdout:Total 5.5 MB/s | 5.8 MB 00:01 2023-12-25T20:50:40.748 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-12-25T20:50:40.757 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-12-25T20:50:40.758 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-12-25T20:50:40.856 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-12-25T20:50:40.859 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-12-25T20:50:40.957 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-25T20:50:41.172 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-12-25T20:50:41.419 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-12-25T20:50:41.688 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-12-25T20:50:42.209 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2023-12-25T20:50:42.402 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-385.gb14a0a1 8/8 2023-12-25T20:50:42.603 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-12-25T20:50:42.726 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-385.gb14a0a1 8/8 2023-12-25T20:50:42.726 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-385.gb14a0a1 1/8 2023-12-25T20:50:42.726 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-12-25T20:50:42.726 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-12-25T20:50:42.726 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-12-25T20:50:42.726 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-12-25T20:50:42.726 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2023-12-25T20:50:42.726 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-12-25T20:50:42.980 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-12-25T20:50:43.270 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-12-25T20:50:43.428 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-385.gb14a0a10.el8.noarch 6/6 2023-12-25T20:50:43.456 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-12-25T20:50:43.456 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:43.456 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-12-25T20:50:43.457 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:16.2.14-385.gb14a0a10.el8.noarch 2023-12-25T20:50:43.457 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-12-25T20:50:43.457 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-12-25T20:50:43.457 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.15-4.el8.x86_64 2023-12-25T20:50:43.457 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-12-25T20:50:43.457 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-12-25T20:50:43.457 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-12-25T20:50:43.457 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2023-12-25T20:50:43.457 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:43.457 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:50:43.576 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-rook 2023-12-25T20:50:43.711 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-385.gb14a0a10.el8.noarch 6/6 2023-12-25T20:50:43.711 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-385.gb14a0a10.el8.noarch 1/6 2023-12-25T20:50:43.711 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-12-25T20:50:43.711 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-12-25T20:50:43.712 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-12-25T20:50:43.712 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-12-25T20:50:44.409 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-12-25T20:50:44.410 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:44.410 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-12-25T20:50:44.410 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm-2:16.2.14-385.gb14a0a10.el8.noarch 2023-12-25T20:50:44.410 INFO:teuthology.orchestra.run.smithi191.stdout: python3-apipkg-1.5-6.el8.noarch 2023-12-25T20:50:44.410 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel-2.5.1-7.el8.noarch 2023-12-25T20:50:44.410 INFO:teuthology.orchestra.run.smithi191.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-12-25T20:50:44.410 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-12-25T20:50:44.410 INFO:teuthology.orchestra.run.smithi191.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-12-25T20:50:44.410 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:44.410 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:50:44.427 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:58 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:50:44.520 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-fuse 2023-12-25T20:50:45.379 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:13 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:50:46.118 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:50:46.120 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:50:46.120 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2023-12-25T20:50:46.120 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:50:46.120 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-12-25T20:50:46.120 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:16.2.14-385.gb14a0a10.el8 ceph-noarch 27 k 2023-12-25T20:50:46.120 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2023-12-25T20:50:46.120 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-25T20:50:46.120 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-25T20:50:46.120 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-25T20:50:46.120 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-12-25T20:50:46.121 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-12-25T20:50:46.121 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-25T20:50:46.121 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-12-25T20:50:46.121 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-12-25T20:50:46.121 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-12-25T20:50:46.121 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-12-25T20:50:46.121 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-25T20:50:46.121 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-25T20:50:46.121 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:46.121 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-12-25T20:50:46.121 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:50:46.121 INFO:teuthology.orchestra.run.smithi153.stdout:Install 13 Packages 2023-12-25T20:50:46.121 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:46.122 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 1.9 M 2023-12-25T20:50:46.122 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 24 M 2023-12-25T20:50:46.122 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-12-25T20:50:46.918 INFO:teuthology.orchestra.run.smithi153.stdout:(1/13): ceph-mgr-rook-16.2.14-385.gb14a0a10.el8 180 kB/s | 27 kB 00:00 2023-12-25T20:50:47.018 INFO:teuthology.orchestra.run.smithi153.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 105 kB/s | 27 kB 00:00 2023-12-25T20:50:47.044 INFO:teuthology.orchestra.run.smithi153.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 72 kB/s | 20 kB 00:00 2023-12-25T20:50:47.066 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:50:47.067 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:50:47.067 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2023-12-25T20:50:47.067 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:50:47.067 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-12-25T20:50:47.067 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 911 k 2023-12-25T20:50:47.067 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2023-12-25T20:50:47.067 INFO:teuthology.orchestra.run.smithi191.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-12-25T20:50:47.068 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:47.068 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-12-25T20:50:47.068 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:50:47.068 INFO:teuthology.orchestra.run.smithi191.stdout:Install 2 Packages 2023-12-25T20:50:47.068 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:47.068 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 994 k 2023-12-25T20:50:47.068 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 2.8 M 2023-12-25T20:50:47.069 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-12-25T20:50:47.194 INFO:teuthology.orchestra.run.smithi153.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 627 kB/s | 110 kB 00:00 2023-12-25T20:50:47.236 INFO:teuthology.orchestra.run.smithi153.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 807 kB/s | 155 kB 00:00 2023-12-25T20:50:47.345 INFO:teuthology.orchestra.run.smithi153.stdout:(6/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 297 kB/s | 126 kB 00:00 2023-12-25T20:50:47.387 INFO:teuthology.orchestra.run.smithi153.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 220 kB/s | 33 kB 00:00 2023-12-25T20:50:47.412 INFO:teuthology.orchestra.run.smithi153.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 196 kB/s | 43 kB 00:00 2023-12-25T20:50:47.497 INFO:teuthology.orchestra.run.smithi153.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 770 kB/s | 81 kB 00:00 2023-12-25T20:50:47.506 INFO:teuthology.orchestra.run.smithi191.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 358 kB/s | 83 kB 00:00 2023-12-25T20:50:47.529 INFO:teuthology.orchestra.run.smithi153.stdout:(10/13): python3-certifi-2018.10.15-7.el8.noarc 88 kB/s | 16 kB 00:00 2023-12-25T20:50:47.555 INFO:teuthology.orchestra.run.smithi153.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 1.1 MB/s | 64 kB 00:00 2023-12-25T20:50:47.596 INFO:teuthology.orchestra.run.smithi153.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 917 kB/s | 61 kB 00:00 2023-12-25T20:50:47.697 INFO:teuthology.orchestra.run.smithi153.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.1 MB/s | 1.2 MB 00:00 2023-12-25T20:50:47.699 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:50:47.699 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.2 MB/s | 1.9 MB 00:01 2023-12-25T20:50:47.759 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-12-25T20:50:47.775 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-12-25T20:50:47.775 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-12-25T20:50:47.863 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-12-25T20:50:47.865 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-12-25T20:50:48.223 INFO:teuthology.orchestra.run.smithi191.stdout:(2/2): ceph-fuse-16.2.14-385.gb14a0a10.el8.x86_ 961 kB/s | 911 kB 00:00 2023-12-25T20:50:48.223 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:50:48.224 INFO:teuthology.orchestra.run.smithi191.stdout:Total 860 kB/s | 994 kB 00:01 2023-12-25T20:50:48.235 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-12-25T20:50:48.256 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-12-25T20:50:48.256 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-12-25T20:50:48.343 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-12-25T20:50:48.346 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-12-25T20:50:48.368 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-12-25T20:50:48.514 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-12-25T20:50:48.587 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-25T20:50:48.775 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-12-25T20:50:48.776 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-12-25T20:50:49.002 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-fuse-2:16.2.14-385.gb14a0a10.el8.x86_64 2/2 2023-12-25T20:50:49.003 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-12-25T20:50:49.216 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-12-25T20:50:49.389 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-fuse-2:16.2.14-385.gb14a0a10.el8.x86_64 2/2 2023-12-25T20:50:49.389 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-fuse-2:16.2.14-385.gb14a0a10.el8.x86_64 1/2 2023-12-25T20:50:49.431 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-12-25T20:50:49.679 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-12-25T20:50:49.968 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-12-25T20:50:50.102 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-12-25T20:50:50.102 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:50.102 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-12-25T20:50:50.102 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse-2:16.2.14-385.gb14a0a10.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-12-25T20:50:50.102 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:50.102 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:50:50.230 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-12-25T20:50:50.294 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librados-devel 2023-12-25T20:50:51.169 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:19 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:50:51.279 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-12-25T20:50:51.524 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-12-25T20:50:51.740 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-12-25T20:50:51.925 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-12-25T20:50:52.084 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:16.2.14-385.gb14a0a10.el8.noarch 13/13 2023-12-25T20:50:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-385.gb14a0a10.el8.noarch 13/13 2023-12-25T20:50:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:16.2.14-385.gb14a0a10.el8.noarch 1/13 2023-12-25T20:50:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-12-25T20:50:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-12-25T20:50:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-12-25T20:50:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-12-25T20:50:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-12-25T20:50:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-12-25T20:50:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-12-25T20:50:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-25T20:50:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-12-25T20:50:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-12-25T20:50:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-12-25T20:50:52.864 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:50:52.865 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:50:52.865 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2023-12-25T20:50:52.865 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:50:52.865 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-12-25T20:50:52.865 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 131 k 2023-12-25T20:50:52.865 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:52.865 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-12-25T20:50:52.865 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:50:52.865 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2023-12-25T20:50:52.865 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:52.866 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 131 k 2023-12-25T20:50:52.866 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 432 k 2023-12-25T20:50:52.866 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-12-25T20:50:53.112 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-12-25T20:50:53.112 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:53.112 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-12-25T20:50:53.112 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:16.2.14-385.gb14a0a10.el8.noarch 2023-12-25T20:50:53.112 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-25T20:50:53.112 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-25T20:50:53.112 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-25T20:50:53.113 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-25T20:50:53.113 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-25T20:50:53.113 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-25T20:50:53.113 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-25T20:50:53.113 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-25T20:50:53.113 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-25T20:50:53.113 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-25T20:50:53.113 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-25T20:50:53.113 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-25T20:50:53.113 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:53.113 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:50:53.236 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-cephadm 2023-12-25T20:50:53.313 INFO:teuthology.orchestra.run.smithi191.stdout:librados-devel-16.2.14-385.gb14a0a10.el8.x86_64 293 kB/s | 131 kB 00:00 2023-12-25T20:50:53.314 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:50:53.314 INFO:teuthology.orchestra.run.smithi191.stdout:Total 292 kB/s | 131 kB 00:00 2023-12-25T20:50:53.315 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-12-25T20:50:53.320 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-12-25T20:50:53.320 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-12-25T20:50:53.376 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-12-25T20:50:53.378 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-12-25T20:50:53.968 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-12-25T20:50:54.088 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:08 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:50:55.342 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librados-devel-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:50:55.646 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados-devel-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:50:55.756 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:50:55.757 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:50:55.757 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2023-12-25T20:50:55.757 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:50:55.757 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-12-25T20:50:55.757 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:16.2.14-385.gb14a0a10.el8 ceph-noarch 116 k 2023-12-25T20:50:55.757 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2023-12-25T20:50:55.758 INFO:teuthology.orchestra.run.smithi153.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-25T20:50:55.758 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-12-25T20:50:55.758 INFO:teuthology.orchestra.run.smithi153.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-25T20:50:55.758 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-12-25T20:50:55.758 INFO:teuthology.orchestra.run.smithi153.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-25T20:50:55.758 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:55.758 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-12-25T20:50:55.758 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:50:55.758 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2023-12-25T20:50:55.758 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:50:55.759 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 5.8 M 2023-12-25T20:50:55.759 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 24 M 2023-12-25T20:50:55.759 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-12-25T20:50:56.179 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): ceph-mgr-cephadm-16.2.14-385.gb14a0a10.e 600 kB/s | 116 kB 00:00 2023-12-25T20:50:56.316 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados-devel-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:50:56.316 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:56.316 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-12-25T20:50:56.316 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:50:56.317 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:50:56.317 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:50:56.516 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install libcephfs2 2023-12-25T20:50:56.537 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 975 kB/s | 538 kB 00:00 2023-12-25T20:50:56.563 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 55 kB/s | 21 kB 00:00 2023-12-25T20:50:56.754 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 255 kB/s | 49 kB 00:00 2023-12-25T20:50:56.905 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): python3-babel-2.5.1-7.el8.noarch.rpm 5.2 MB/s | 4.8 MB 00:00 2023-12-25T20:50:57.149 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 581 kB/s | 355 kB 00:00 2023-12-25T20:50:57.151 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:50:57.151 INFO:teuthology.orchestra.run.smithi153.stdout:Total 4.2 MB/s | 5.8 MB 00:01 2023-12-25T20:50:57.207 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-12-25T20:50:57.217 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-12-25T20:50:57.217 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-12-25T20:50:57.319 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-12-25T20:50:57.322 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-12-25T20:50:57.378 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:25 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:50:57.632 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-12-25T20:50:57.933 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-12-25T20:50:58.178 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-12-25T20:50:58.820 INFO:teuthology.orchestra.run.smithi191.stdout:Package libcephfs2-2:16.2.14-385.gb14a0a10.el8.x86_64 is already installed. 2023-12-25T20:50:59.043 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:50:59.045 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2023-12-25T20:50:59.045 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:50:59.102 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-12-25T20:50:59.141 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install libcephfs-devel 2023-12-25T20:50:59.488 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-12-25T20:50:59.732 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-12-25T20:50:59.907 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-385.gb14a0a10.el8.noarch 6/6 2023-12-25T20:51:00.003 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:28 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:51:00.226 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-385.gb14a0a10.el8.noarch 6/6 2023-12-25T20:51:00.226 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-385.gb14a0a10.el8.noarch 1/6 2023-12-25T20:51:00.227 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-12-25T20:51:00.227 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-12-25T20:51:00.227 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-12-25T20:51:00.227 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-12-25T20:51:01.036 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-12-25T20:51:01.036 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:01.036 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-12-25T20:51:01.036 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:16.2.14-385.gb14a0a10.el8.noarch 2023-12-25T20:51:01.036 INFO:teuthology.orchestra.run.smithi153.stdout: python3-apipkg-1.5-6.el8.noarch 2023-12-25T20:51:01.036 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.5.1-7.el8.noarch 2023-12-25T20:51:01.036 INFO:teuthology.orchestra.run.smithi153.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-12-25T20:51:01.036 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-12-25T20:51:01.036 INFO:teuthology.orchestra.run.smithi153.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-12-25T20:51:01.036 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:01.037 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:51:01.145 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-fuse 2023-12-25T20:51:01.691 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:51:01.691 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:51:01.692 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2023-12-25T20:51:01.692 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:51:01.692 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-12-25T20:51:01.692 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 25 k 2023-12-25T20:51:01.692 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:01.692 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-12-25T20:51:01.692 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:51:01.692 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2023-12-25T20:51:01.692 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:01.692 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 25 k 2023-12-25T20:51:01.692 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 107 k 2023-12-25T20:51:01.693 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-12-25T20:51:01.856 INFO:teuthology.orchestra.run.smithi191.stdout:libcephfs-devel-16.2.14-385.gb14a0a10.el8.x86_6 156 kB/s | 25 kB 00:00 2023-12-25T20:51:01.857 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:51:01.857 INFO:teuthology.orchestra.run.smithi191.stdout:Total 154 kB/s | 25 kB 00:00 2023-12-25T20:51:01.857 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-12-25T20:51:01.860 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-12-25T20:51:01.861 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-12-25T20:51:01.863 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-12-25T20:51:01.866 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-12-25T20:51:01.945 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-12-25T20:51:02.005 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:16 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:51:02.132 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs-devel-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:02.399 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:03.070 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs-devel-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:03.070 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:03.070 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-12-25T20:51:03.070 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:51:03.070 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:03.070 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:51:03.184 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librados2 2023-12-25T20:51:03.692 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:51:03.692 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:51:03.693 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2023-12-25T20:51:03.693 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:51:03.693 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-12-25T20:51:03.693 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 911 k 2023-12-25T20:51:03.693 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2023-12-25T20:51:03.693 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-12-25T20:51:03.693 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:03.693 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-12-25T20:51:03.693 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:51:03.693 INFO:teuthology.orchestra.run.smithi153.stdout:Install 2 Packages 2023-12-25T20:51:03.693 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:03.694 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 994 k 2023-12-25T20:51:03.694 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 2.8 M 2023-12-25T20:51:03.694 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-12-25T20:51:03.965 INFO:teuthology.orchestra.run.smithi153.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 451 kB/s | 83 kB 00:00 2023-12-25T20:51:04.039 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:32 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:51:04.382 INFO:teuthology.orchestra.run.smithi153.stdout:(2/2): ceph-fuse-16.2.14-385.gb14a0a10.el8.x86_ 1.5 MB/s | 911 kB 00:00 2023-12-25T20:51:04.382 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:51:04.383 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.4 MB/s | 994 kB 00:00 2023-12-25T20:51:04.393 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-12-25T20:51:04.414 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-12-25T20:51:04.414 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-12-25T20:51:04.500 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-12-25T20:51:04.503 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-12-25T20:51:04.708 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-12-25T20:51:04.981 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-12-25T20:51:05.165 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:16.2.14-385.gb14a0a10.el8.x86_64 2/2 2023-12-25T20:51:05.502 INFO:teuthology.orchestra.run.smithi191.stdout:Package librados2-2:16.2.14-385.gb14a0a10.el8.x86_64 is already installed. 2023-12-25T20:51:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-fuse-2:16.2.14-385.gb14a0a10.el8.x86_64 2/2 2023-12-25T20:51:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:16.2.14-385.gb14a0a10.el8.x86_64 1/2 2023-12-25T20:51:05.728 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:51:05.729 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2023-12-25T20:51:05.729 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:51:05.979 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librbd1 2023-12-25T20:51:06.257 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-12-25T20:51:06.257 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:06.258 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-12-25T20:51:06.258 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:16.2.14-385.gb14a0a10.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-12-25T20:51:06.258 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:06.258 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:51:06.463 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados-devel 2023-12-25T20:51:06.835 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:34 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:51:07.333 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:21 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:51:08.289 INFO:teuthology.orchestra.run.smithi191.stdout:Package librbd1-2:16.2.14-385.gb14a0a10.el8.x86_64 is already installed. 2023-12-25T20:51:08.520 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:51:08.521 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2023-12-25T20:51:08.521 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:51:08.613 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rados 2023-12-25T20:51:09.022 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:51:09.022 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:51:09.023 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2023-12-25T20:51:09.023 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:51:09.023 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-12-25T20:51:09.023 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 131 k 2023-12-25T20:51:09.023 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:09.023 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-12-25T20:51:09.023 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:51:09.023 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2023-12-25T20:51:09.023 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:09.024 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 131 k 2023-12-25T20:51:09.024 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 432 k 2023-12-25T20:51:09.024 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-12-25T20:51:09.410 INFO:teuthology.orchestra.run.smithi153.stdout:librados-devel-16.2.14-385.gb14a0a10.el8.x86_64 339 kB/s | 131 kB 00:00 2023-12-25T20:51:09.410 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:51:09.411 INFO:teuthology.orchestra.run.smithi153.stdout:Total 338 kB/s | 131 kB 00:00 2023-12-25T20:51:09.411 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-12-25T20:51:09.416 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-12-25T20:51:09.417 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-12-25T20:51:09.466 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-12-25T20:51:09.470 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-12-25T20:51:09.474 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:37 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:51:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-12-25T20:51:09.788 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:10.081 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados-devel-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:10.750 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:10.750 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:10.750 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-12-25T20:51:10.750 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:51:10.750 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:10.750 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:51:10.934 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rados-2:16.2.14-385.gb14a0a10.el8.x86_64 is already installed. 2023-12-25T20:51:10.965 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs2 2023-12-25T20:51:11.157 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:51:11.159 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2023-12-25T20:51:11.159 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:51:11.248 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rgw 2023-12-25T20:51:11.847 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:25 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:51:12.102 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:40 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:51:13.425 INFO:teuthology.orchestra.run.smithi153.stdout:Package libcephfs2-2:16.2.14-385.gb14a0a10.el8.x86_64 is already installed. 2023-12-25T20:51:13.560 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rgw-2:16.2.14-385.gb14a0a10.el8.x86_64 is already installed. 2023-12-25T20:51:13.650 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:51:13.651 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2023-12-25T20:51:13.651 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:51:13.749 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs-devel 2023-12-25T20:51:13.787 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:51:13.788 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2023-12-25T20:51:13.788 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:51:13.885 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-cephfs 2023-12-25T20:51:14.614 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:28 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:51:14.734 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:42 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:51:16.187 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-cephfs-2:16.2.14-385.gb14a0a10.el8.x86_64 is already installed. 2023-12-25T20:51:16.309 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:51:16.309 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:51:16.309 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2023-12-25T20:51:16.309 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:51:16.309 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-12-25T20:51:16.310 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 25 k 2023-12-25T20:51:16.310 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:16.310 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-12-25T20:51:16.310 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:51:16.310 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2023-12-25T20:51:16.310 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:16.310 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 25 k 2023-12-25T20:51:16.310 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 107 k 2023-12-25T20:51:16.310 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-12-25T20:51:16.413 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:51:16.414 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2023-12-25T20:51:16.414 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:51:16.458 INFO:teuthology.orchestra.run.smithi153.stdout:libcephfs-devel-16.2.14-385.gb14a0a10.el8.x86_6 170 kB/s | 25 kB 00:00 2023-12-25T20:51:16.458 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:51:16.459 INFO:teuthology.orchestra.run.smithi153.stdout:Total 168 kB/s | 25 kB 00:00 2023-12-25T20:51:16.459 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-12-25T20:51:16.462 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-12-25T20:51:16.463 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-12-25T20:51:16.465 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-12-25T20:51:16.468 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-12-25T20:51:16.516 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-12-25T20:51:16.660 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rbd 2023-12-25T20:51:16.746 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:17.013 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:17.531 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:45 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:51:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:17.687 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-12-25T20:51:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:51:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:17.687 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:51:17.794 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados2 2023-12-25T20:51:18.649 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:32 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:51:18.997 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rbd-2:16.2.14-385.gb14a0a10.el8.x86_64 is already installed. 2023-12-25T20:51:19.223 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:51:19.224 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2023-12-25T20:51:19.224 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:51:19.320 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-fuse 2023-12-25T20:51:20.122 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:16.2.14-385.gb14a0a10.el8.x86_64 is already installed. 2023-12-25T20:51:20.181 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:48 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:51:20.349 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:51:20.350 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2023-12-25T20:51:20.350 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:51:20.443 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librbd1 2023-12-25T20:51:21.303 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:35 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:51:21.868 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:51:21.869 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:51:21.869 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2023-12-25T20:51:21.869 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:51:21.869 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-12-25T20:51:21.869 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 88 k 2023-12-25T20:51:21.869 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:21.869 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-12-25T20:51:21.869 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:51:21.870 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2023-12-25T20:51:21.870 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:21.870 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 88 k 2023-12-25T20:51:21.870 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 240 k 2023-12-25T20:51:21.870 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-12-25T20:51:22.032 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-fuse-16.2.14-385.gb14a0a10.el8.x86_64.rpm 547 kB/s | 88 kB 00:00 2023-12-25T20:51:22.032 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:51:22.032 INFO:teuthology.orchestra.run.smithi191.stdout:Total 542 kB/s | 88 kB 00:00 2023-12-25T20:51:22.033 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-12-25T20:51:22.042 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-12-25T20:51:22.042 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-12-25T20:51:22.091 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-12-25T20:51:22.094 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-12-25T20:51:22.218 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-12-25T20:51:22.403 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-fuse-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:22.698 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-fuse-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:22.774 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:16.2.14-385.gb14a0a10.el8.x86_64 is already installed. 2023-12-25T20:51:22.999 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:51:23.000 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2023-12-25T20:51:23.000 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:51:23.105 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rados 2023-12-25T20:51:23.407 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-fuse-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:23.407 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:23.407 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-12-25T20:51:23.407 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:51:23.407 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:23.407 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:51:23.637 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-mirror 2023-12-25T20:51:23.967 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:37 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:51:24.500 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:52 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:51:25.422 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rados-2:16.2.14-385.gb14a0a10.el8.x86_64 is already installed. 2023-12-25T20:51:25.649 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:51:25.650 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2023-12-25T20:51:25.650 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:51:25.755 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rgw 2023-12-25T20:51:26.192 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:51:26.193 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:51:26.193 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2023-12-25T20:51:26.193 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:51:26.193 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-12-25T20:51:26.193 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 3.7 M 2023-12-25T20:51:26.193 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:26.193 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-12-25T20:51:26.193 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:51:26.193 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2023-12-25T20:51:26.193 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:26.194 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 3.7 M 2023-12-25T20:51:26.194 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 15 M 2023-12-25T20:51:26.194 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-12-25T20:51:26.615 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:40 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:51:28.078 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rgw-2:16.2.14-385.gb14a0a10.el8.x86_64 is already installed. 2023-12-25T20:51:28.305 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:51:28.306 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2023-12-25T20:51:28.306 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:51:28.432 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-cephfs 2023-12-25T20:51:28.538 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-mirror-16.2.14-385.gb14a0a10.el8.x86_64.rpm 1.6 MB/s | 3.7 MB 00:02 2023-12-25T20:51:28.539 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:51:28.539 INFO:teuthology.orchestra.run.smithi191.stdout:Total 1.6 MB/s | 3.7 MB 00:02 2023-12-25T20:51:28.539 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-12-25T20:51:28.552 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-12-25T20:51:28.553 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-12-25T20:51:28.621 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-12-25T20:51:28.624 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-12-25T20:51:29.059 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-12-25T20:51:29.254 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-mirror-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:29.288 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:43 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:51:29.690 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-mirror-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:30.415 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-mirror-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:30.415 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:30.415 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-12-25T20:51:30.416 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:51:30.416 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:30.416 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:51:30.645 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-nbd 2023-12-25T20:51:30.750 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-cephfs-2:16.2.14-385.gb14a0a10.el8.x86_64 is already installed. 2023-12-25T20:51:30.980 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:51:30.981 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2023-12-25T20:51:30.981 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:51:31.077 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rbd 2023-12-25T20:51:31.509 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:59 ago on Mon 25 Dec 2023 08:48:32 PM UTC. 2023-12-25T20:51:31.944 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:45 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:51:33.187 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-25T20:51:33.187 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:51:33.188 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2023-12-25T20:51:33.188 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:51:33.188 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-12-25T20:51:33.188 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 175 k 2023-12-25T20:51:33.188 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:33.188 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-12-25T20:51:33.188 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-25T20:51:33.188 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2023-12-25T20:51:33.188 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:33.189 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 175 k 2023-12-25T20:51:33.189 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 523 k 2023-12-25T20:51:33.189 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-12-25T20:51:33.415 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rbd-2:16.2.14-385.gb14a0a10.el8.x86_64 is already installed. 2023-12-25T20:51:33.641 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:51:33.642 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2023-12-25T20:51:33.642 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:51:33.742 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-fuse 2023-12-25T20:51:33.752 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-nbd-16.2.14-385.gb14a0a10.el8.x86_64.rpm 311 kB/s | 175 kB 00:00 2023-12-25T20:51:33.752 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:51:33.753 INFO:teuthology.orchestra.run.smithi191.stdout:Total 310 kB/s | 175 kB 00:00 2023-12-25T20:51:33.753 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-12-25T20:51:33.764 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-12-25T20:51:33.764 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-12-25T20:51:33.815 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-12-25T20:51:33.817 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-12-25T20:51:33.934 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-12-25T20:51:34.126 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-nbd-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:34.425 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-nbd-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:34.597 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:48 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:51:35.133 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-nbd-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:35.133 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:35.134 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-12-25T20:51:35.134 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:51:35.134 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:35.134 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-25T20:51:35.377 DEBUG:teuthology.parallel:result is None 2023-12-25T20:51:36.274 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:51:36.274 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:51:36.274 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2023-12-25T20:51:36.274 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:51:36.274 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-12-25T20:51:36.274 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 88 k 2023-12-25T20:51:36.274 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:36.275 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-12-25T20:51:36.275 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:51:36.275 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2023-12-25T20:51:36.275 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:36.275 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 88 k 2023-12-25T20:51:36.275 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 240 k 2023-12-25T20:51:36.276 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-12-25T20:51:36.664 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-fuse-16.2.14-385.gb14a0a10.el8.x86_64.rpm 227 kB/s | 88 kB 00:00 2023-12-25T20:51:36.665 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:51:36.666 INFO:teuthology.orchestra.run.smithi153.stdout:Total 227 kB/s | 88 kB 00:00 2023-12-25T20:51:36.666 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-12-25T20:51:36.675 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-12-25T20:51:36.675 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-12-25T20:51:36.723 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-12-25T20:51:36.726 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-12-25T20:51:36.832 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-12-25T20:51:37.042 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:37.333 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-fuse-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:38.021 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:38.021 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:38.022 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-12-25T20:51:38.022 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:51:38.022 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:38.022 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:51:38.213 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-mirror 2023-12-25T20:51:39.079 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:53 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:51:40.767 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:51:40.800 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:51:40.800 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2023-12-25T20:51:40.800 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:51:40.800 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-12-25T20:51:40.800 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 3.7 M 2023-12-25T20:51:40.800 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:40.800 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-12-25T20:51:40.800 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:51:40.800 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2023-12-25T20:51:40.800 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:40.801 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 3.7 M 2023-12-25T20:51:40.801 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 15 M 2023-12-25T20:51:40.801 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-12-25T20:51:42.300 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-mirror-16.2.14-385.gb14a0a10.el8.x86_64.rpm 2.4 MB/s | 3.7 MB 00:01 2023-12-25T20:51:42.301 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:51:42.301 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.4 MB/s | 3.7 MB 00:01 2023-12-25T20:51:42.301 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-12-25T20:51:42.314 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-12-25T20:51:42.314 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-12-25T20:51:42.382 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-12-25T20:51:42.386 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-12-25T20:51:42.822 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-12-25T20:51:43.001 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:43.449 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:44.153 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:44.153 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:44.153 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-12-25T20:51:44.153 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:51:44.153 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:44.153 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:51:44.342 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-nbd 2023-12-25T20:51:45.205 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:59 ago on Mon 25 Dec 2023 08:48:46 PM UTC. 2023-12-25T20:51:46.891 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-12-25T20:51:46.892 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:51:46.892 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2023-12-25T20:51:46.892 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:51:46.892 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-12-25T20:51:46.892 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:16.2.14-385.gb14a0a10.el8 ceph 175 k 2023-12-25T20:51:46.892 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:46.892 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-12-25T20:51:46.892 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-12-25T20:51:46.892 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2023-12-25T20:51:46.892 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:46.893 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 175 k 2023-12-25T20:51:46.893 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 523 k 2023-12-25T20:51:46.893 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-12-25T20:51:47.294 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-nbd-16.2.14-385.gb14a0a10.el8.x86_64.rpm 437 kB/s | 175 kB 00:00 2023-12-25T20:51:47.294 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:51:47.294 INFO:teuthology.orchestra.run.smithi153.stdout:Total 436 kB/s | 175 kB 00:00 2023-12-25T20:51:47.295 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-12-25T20:51:47.305 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-12-25T20:51:47.306 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-12-25T20:51:47.357 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-12-25T20:51:47.360 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-12-25T20:51:47.474 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-12-25T20:51:47.657 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:47.956 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-nbd-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:48.649 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:16.2.14-385.gb14a0a10.el8.x86_64 1/1 2023-12-25T20:51:48.649 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:48.649 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-12-25T20:51:48.649 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:16.2.14-385.gb14a0a10.el8.x86_64 2023-12-25T20:51:48.649 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:48.649 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-12-25T20:51:48.827 DEBUG:teuthology.parallel:result is None 2023-12-25T20:51:48.828 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 2023-12-25T20:51:49.042 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-25T20:51:49.064 INFO:teuthology.orchestra.run.smithi153.stdout:16.2.14-385.gb14a0a10.el8 2023-12-25T20:51:49.065 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-385.gb14a0a10.el8 2023-12-25T20:51:49.065 INFO:teuthology.task.install:The correct ceph version 16.2.14-385.gb14a0a10 is installed. 2023-12-25T20:51:49.067 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 2023-12-25T20:51:49.289 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-25T20:51:49.316 INFO:teuthology.orchestra.run.smithi191.stdout:16.2.14-385.gb14a0a10.el8 2023-12-25T20:51:49.316 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-385.gb14a0a10.el8 2023-12-25T20:51:49.316 INFO:teuthology.task.install:The correct ceph version 16.2.14-385.gb14a0a10 is installed. 2023-12-25T20:51:49.318 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-25T20:51:49.318 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-12-25T20:51:49.318 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-25T20:51:49.345 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-25T20:51:49.346 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-25T20:51:49.390 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-25T20:51:49.390 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-12-25T20:51:49.390 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2023-12-25T20:51:49.417 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-25T20:51:49.483 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-25T20:51:49.483 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/daemon-helper 2023-12-25T20:51:49.514 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-25T20:51:49.585 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-25T20:51:49.585 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-12-25T20:51:49.585 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-25T20:51:49.612 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-25T20:51:49.679 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-25T20:51:49.679 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-25T20:51:49.710 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-25T20:51:49.779 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-25T20:51:49.780 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-12-25T20:51:49.780 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2023-12-25T20:51:49.806 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-25T20:51:49.873 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-25T20:51:49.873 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/stdin-killer 2023-12-25T20:51:49.901 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-25T20:51:49.970 INFO:teuthology.run_tasks:Running task ceph... 2023-12-25T20:51:50.044 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-25T20:51:50.044 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /var/log/ceph 2023-12-25T20:51:50.047 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /var/log/ceph 2023-12-25T20:51:50.074 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-25T20:51:50.074 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-25T20:51:50.114 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-25T20:51:50.141 INFO:tasks.ceph:Creating extra log directories... 2023-12-25T20:51:50.141 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-25T20:51:50.181 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-25T20:51:50.219 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-25T20:51:50.219 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-25T20:51:50.219 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500952', 'branch': 'wip-yuri5-testing-2023-12-15-0747-pacific', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/one workloads/rados_mon_workunits}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7500952', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri5-testing-2023-12-15-0747-pacific'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'sha1': 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri5-testing-2023-12-15-0747-pacific', 'sha1': 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 2834, 'sha1': 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri5-testing-2023-12-15-0747-pacific', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2', 'targets': {'smithi153.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHwsX+5nIByVAfAQ8I7l1bnFaNLiDHwJYt5U5Zvs8zKhl42cGlAb4liT5eIqnXZhTjMK80xEHH1sp+lHie9P4w4=', 'smithi191.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD+ISzGpHk1nnk50JTFdMJHaU3XrYxVp8J6UrAV6J4AP0mz5orSF5TRpCHUvF8HD2drq6QTywF6azXOm3Jow5dE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 20, 'thrash_delay': 1}}, {'workunit': {'clients': {'client.0': ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9680d2aba1e76f64ade295c81599c97907457848', 'timestamp': '2023-12-25_17:28:36', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078'} 2023-12-25T20:51:50.220 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-25T20:51:50.248 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-25T20:51:50.267 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/run/ceph 2023-12-25T20:51:50.306 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/run/ceph 2023-12-25T20:51:50.334 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-12-25T20:51:50.334 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2023-12-25T20:51:50.388 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-25T20:51:50.388 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2023-12-25T20:51:50.446 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-25T20:51:50.446 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-25T20:51:50.446 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 6h/6d Inode: 389810 Links: 1 2023-12-25T20:51:50.446 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-25T20:51:50.447 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2023-12-25T20:51:50.447 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2023-12-25 20:51:43.199893320 +0000 2023-12-25T20:51:50.447 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2023-12-25 20:47:07.894247821 +0000 2023-12-25T20:51:50.447 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2023-12-25 20:47:07.894247821 +0000 2023-12-25T20:51:50.447 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2023-12-25T20:51:50.447 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-25T20:51:50.511 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2023-12-25T20:51:50.512 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2023-12-25T20:51:50.512 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000236942 s, 2.2 MB/s 2023-12-25T20:51:50.512 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-25T20:51:50.571 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2023-12-25T20:51:50.629 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-25T20:51:50.629 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-25T20:51:50.629 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 6h/6d Inode: 395422 Links: 1 2023-12-25T20:51:50.629 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-25T20:51:50.630 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2023-12-25T20:51:50.630 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2023-12-25 20:51:43.199893320 +0000 2023-12-25T20:51:50.630 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2023-12-25 20:47:08.393239260 +0000 2023-12-25T20:51:50.630 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2023-12-25 20:47:08.393239260 +0000 2023-12-25T20:51:50.630 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2023-12-25T20:51:50.630 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-25T20:51:50.694 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2023-12-25T20:51:50.694 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2023-12-25T20:51:50.694 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000213172 s, 2.4 MB/s 2023-12-25T20:51:50.695 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-25T20:51:50.754 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2023-12-25T20:51:50.812 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-25T20:51:50.812 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-25T20:51:50.812 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 6h/6d Inode: 389943 Links: 1 2023-12-25T20:51:50.812 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-25T20:51:50.813 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2023-12-25T20:51:50.813 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2023-12-25 20:51:43.199893320 +0000 2023-12-25T20:51:50.813 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2023-12-25 20:47:08.906230459 +0000 2023-12-25T20:51:50.813 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2023-12-25 20:47:08.906230459 +0000 2023-12-25T20:51:50.813 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2023-12-25T20:51:50.813 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-25T20:51:50.877 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2023-12-25T20:51:50.877 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2023-12-25T20:51:50.877 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000238213 s, 2.1 MB/s 2023-12-25T20:51:50.878 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-25T20:51:50.937 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2023-12-25T20:51:50.994 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-25T20:51:50.994 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-25T20:51:50.994 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 6h/6d Inode: 395637 Links: 1 2023-12-25T20:51:50.994 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-25T20:51:50.994 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2023-12-25T20:51:50.995 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2023-12-25 20:51:43.200893303 +0000 2023-12-25T20:51:50.995 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2023-12-25 20:47:09.403221933 +0000 2023-12-25T20:51:50.995 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2023-12-25 20:47:09.403221933 +0000 2023-12-25T20:51:50.995 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2023-12-25T20:51:50.995 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-25T20:51:51.060 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2023-12-25T20:51:51.060 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2023-12-25T20:51:51.060 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000251027 s, 2.0 MB/s 2023-12-25T20:51:51.061 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-25T20:51:51.120 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-25T20:51:51.120 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-25T20:51:51.120 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2023-12-25T20:51:51.140 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-25T20:51:51.140 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2023-12-25T20:51:51.200 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-25T20:51:51.201 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-25T20:51:51.201 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 394530 Links: 1 2023-12-25T20:51:51.201 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-25T20:51:51.201 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-12-25T20:51:51.201 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-12-25 20:51:29.444023800 +0000 2023-12-25T20:51:51.201 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-12-25 20:47:07.297423313 +0000 2023-12-25T20:51:51.201 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-12-25 20:47:07.297423313 +0000 2023-12-25T20:51:51.201 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-12-25T20:51:51.201 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-25T20:51:51.270 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-12-25T20:51:51.270 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-12-25T20:51:51.270 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000229612 s, 2.2 MB/s 2023-12-25T20:51:51.271 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-25T20:51:51.333 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2023-12-25T20:51:51.392 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-25T20:51:51.392 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-25T20:51:51.392 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 392822 Links: 1 2023-12-25T20:51:51.392 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-25T20:51:51.392 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-12-25T20:51:51.392 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-12-25 20:51:29.444023800 +0000 2023-12-25T20:51:51.392 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-12-25 20:47:07.804414760 +0000 2023-12-25T20:51:51.392 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-12-25 20:47:07.804414760 +0000 2023-12-25T20:51:51.392 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-12-25T20:51:51.393 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-25T20:51:51.460 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-12-25T20:51:51.460 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-12-25T20:51:51.460 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000222727 s, 2.3 MB/s 2023-12-25T20:51:51.461 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-25T20:51:51.521 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2023-12-25T20:51:51.580 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-25T20:51:51.580 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-25T20:51:51.580 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 392952 Links: 1 2023-12-25T20:51:51.580 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-25T20:51:51.580 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-12-25T20:51:51.580 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-12-25 20:51:29.444023800 +0000 2023-12-25T20:51:51.580 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-12-25 20:47:08.315406140 +0000 2023-12-25T20:51:51.580 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-12-25 20:47:08.315406140 +0000 2023-12-25T20:51:51.580 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-12-25T20:51:51.581 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-25T20:51:51.648 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-12-25T20:51:51.648 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-12-25T20:51:51.648 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000222908 s, 2.3 MB/s 2023-12-25T20:51:51.649 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-25T20:51:51.709 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2023-12-25T20:51:51.768 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-25T20:51:51.769 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-25T20:51:51.769 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 398470 Links: 1 2023-12-25T20:51:51.769 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-25T20:51:51.769 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-12-25T20:51:51.769 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-12-25 20:51:29.444023800 +0000 2023-12-25T20:51:51.769 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-12-25 20:47:08.814397722 +0000 2023-12-25T20:51:51.769 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-12-25 20:47:08.814397722 +0000 2023-12-25T20:51:51.769 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-12-25T20:51:51.769 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-25T20:51:51.835 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-12-25T20:51:51.835 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-12-25T20:51:51.835 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000233273 s, 2.2 MB/s 2023-12-25T20:51:51.836 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-25T20:51:51.895 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-12-25T20:51:51.895 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2023-12-25T20:51:51.895 INFO:tasks.ceph:Generating config... 2023-12-25T20:51:51.897 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-25T20:51:51.897 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-12-25T20:51:51.897 INFO:tasks.ceph:[global] ms inject delay max = 1 2023-12-25T20:51:51.897 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2023-12-25T20:51:51.897 INFO:tasks.ceph:[global] ms inject delay type = mon 2023-12-25T20:51:51.897 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2023-12-25T20:51:51.897 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2023-12-25T20:51:51.897 INFO:tasks.ceph:[global] ms type = async 2023-12-25T20:51:51.897 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-25T20:51:51.897 INFO:tasks.ceph:[mgr] debug monc = 10 2023-12-25T20:51:51.898 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-25T20:51:51.898 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-25T20:51:51.898 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-25T20:51:51.898 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-25T20:51:51.898 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2023-12-25T20:51:51.898 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-12-25T20:51:51.898 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-12-25T20:51:51.898 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-12-25T20:51:51.898 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2023-12-25T20:51:51.898 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2023-12-25T20:51:51.898 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-12-25T20:51:51.898 INFO:tasks.ceph:[mon] paxos service trim min = 5 2023-12-25T20:51:51.898 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-25T20:51:51.898 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-25T20:51:51.898 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-25T20:51:51.899 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-25T20:51:51.899 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-12-25T20:51:51.899 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-12-25T20:51:51.899 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-25T20:51:51.899 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-25T20:51:51.899 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-12-25T20:51:51.899 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-25T20:51:51.899 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-25T20:51:51.899 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-25T20:51:51.899 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-12-25T20:51:51.899 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-12-25T20:51:51.899 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-25T20:51:51.899 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2023-12-25T20:51:51.899 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-25T20:51:51.899 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-12-25T20:51:51.900 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-12-25T20:51:51.900 INFO:tasks.ceph:Setting up mon.a... 2023-12-25T20:51:51.900 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-25T20:51:52.004 INFO:teuthology.orchestra.run.smithi153.stdout:creating /etc/ceph/ceph.keyring 2023-12-25T20:51:52.006 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-25T20:51:52.050 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-25T20:51:52.118 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.153'), ('mon.c', '[v2:172.21.15.153:3301,v1:172.21.15.153:6790]'), ('mon.b', '172.21.15.191')] 2023-12-25T20:51:52.119 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.153,[v2:172.21.15.153:3301,v1:172.21.15.153:6790],172.21.15.191', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-12-25T20:51:52.120 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-12-25T20:51:52.120 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-25T20:51:52.176 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.153 --addv c '[v2:172.21.15.153:3301,v1:172.21.15.153:6790]' --add b 172.21.15.191 --print /home/ubuntu/cephtest/ceph.monmap 2023-12-25T20:51:52.261 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-25T20:51:52.261 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: generated fsid 0ece9abc-d1cb-4514-a255-e770deeb21b9 2023-12-25T20:51:52.261 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 0 2023-12-25T20:51:52.261 INFO:teuthology.orchestra.run.smithi153.stdout:fsid 0ece9abc-d1cb-4514-a255-e770deeb21b9 2023-12-25T20:51:52.261 INFO:teuthology.orchestra.run.smithi153.stdout:last_changed 2023-12-25T20:51:52.261713+0000 2023-12-25T20:51:52.261 INFO:teuthology.orchestra.run.smithi153.stdout:created 2023-12-25T20:51:52.261713+0000 2023-12-25T20:51:52.261 INFO:teuthology.orchestra.run.smithi153.stdout:min_mon_release 0 (unknown) 2023-12-25T20:51:52.261 INFO:teuthology.orchestra.run.smithi153.stdout:election_strategy: 1 2023-12-25T20:51:52.262 INFO:teuthology.orchestra.run.smithi153.stdout:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2023-12-25T20:51:52.262 INFO:teuthology.orchestra.run.smithi153.stdout:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-25T20:51:52.262 INFO:teuthology.orchestra.run.smithi153.stdout:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2023-12-25T20:51:52.262 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-12-25T20:51:52.263 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-25T20:51:52.280 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0ece9abc-d1cb-4514-a255-e770deeb21b9... 2023-12-25T20:51:52.281 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-25T20:51:52.323 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-25T20:51:52.365 INFO:teuthology.orchestra.run.smithi153.stdout:[global] 2023-12-25T20:51:52.365 INFO:teuthology.orchestra.run.smithi153.stdout: chdir = "" 2023-12-25T20:51:52.365 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-25T20:51:52.365 INFO:teuthology.orchestra.run.smithi153.stdout: auth supported = cephx 2023-12-25T20:51:52.365 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.365 INFO:teuthology.orchestra.run.smithi153.stdout: filestore xattr use omap = true 2023-12-25T20:51:52.365 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.365 INFO:teuthology.orchestra.run.smithi153.stdout: mon clock drift allowed = 1.000 2023-12-25T20:51:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: osd crush chooseleaf type = 0 2023-12-25T20:51:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: auth debug = true 2023-12-25T20:51:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on old message = true 2023-12-25T20:51:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on bug = true 2023-12-25T20:51:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-25T20:51:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: mon pg warn max object skew = 0 2023-12-25T20:51:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: # disable pg_autoscaler by default for new pools 2023-12-25T20:51:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-25T20:51:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default size = 2 2023-12-25T20:51:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.367 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow primary affinity = true 2023-12-25T20:51:52.367 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow pg remap = true 2023-12-25T20:51:52.367 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on legacy crush tunables = false 2023-12-25T20:51:52.367 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on crush straw calc version zero = false 2023-12-25T20:51:52.367 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on no sortbitwise = false 2023-12-25T20:51:52.367 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on osd down out interval zero = false 2023-12-25T20:51:52.367 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on too few osds = false 2023-12-25T20:51:52.367 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-25T20:51:52.367 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-25T20:51:52.367 INFO:teuthology.orchestra.run.smithi153.stdout: mon_allow_pool_size_one = true 2023-12-25T20:51:52.367 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.368 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-12-25T20:51:52.368 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.368 INFO:teuthology.orchestra.run.smithi153.stdout: osd default data pool replay window = 5 2023-12-25T20:51:52.368 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.368 INFO:teuthology.orchestra.run.smithi153.stdout: mon allow pool delete = true 2023-12-25T20:51:52.368 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.368 INFO:teuthology.orchestra.run.smithi153.stdout: mon cluster log file level = debug 2023-12-25T20:51:52.368 INFO:teuthology.orchestra.run.smithi153.stdout: debug asserts on shutdown = true 2023-12-25T20:51:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: mon health detail to clog = false 2023-12-25T20:51:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: mon host = "172.21.15.153,[v2:172.21.15.153:3301,v1:172.21.15.153:6790],172.21.15.191" 2023-12-25T20:51:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: mon client directed command retry = 5 2023-12-25T20:51:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: mon election default strategy = 1 2023-12-25T20:51:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject delay max = 1 2023-12-25T20:51:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject delay probability = 0.005 2023-12-25T20:51:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject delay type = mon 2023-12-25T20:51:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject internal delays = 0.002 2023-12-25T20:51:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject socket failures = 2500 2023-12-25T20:51:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: ms type = async 2023-12-25T20:51:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: fsid = 0ece9abc-d1cb-4514-a255-e770deeb21b9 2023-12-25T20:51:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.369 INFO:teuthology.orchestra.run.smithi153.stdout:[osd] 2023-12-25T20:51:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: osd journal size = 100 2023-12-25T20:51:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub load threshold = 5.0 2023-12-25T20:51:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub max interval = 600 2023-12-25T20:51:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: osd recover clone overlap = true 2023-12-25T20:51:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: osd recovery max chunk = 1048576 2023-12-25T20:51:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug shutdown = true 2023-12-25T20:51:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug op order = true 2023-12-25T20:51:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify stray on activate = true 2023-12-25T20:51:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: osd open classes on start = true 2023-12-25T20:51:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug pg log writeout = true 2023-12-25T20:51:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: osd deep scrub update digest min age = 30 2023-12-25T20:51:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: osd map max advance = 10 2023-12-25T20:51:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: journal zero on create = true 2023-12-25T20:51:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: filestore ondisk finisher threads = 3 2023-12-25T20:51:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: filestore apply finisher threads = 3 2023-12-25T20:51:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: bdev debug aio = true 2023-12-25T20:51:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug misdirected ops = true 2023-12-25T20:51:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: bdev async discard = True 2023-12-25T20:51:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: bdev enable discard = True 2023-12-25T20:51:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore block size = 96636764160 2023-12-25T20:51:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore fsck on mount = True 2023-12-25T20:51:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluefs = 20 2023-12-25T20:51:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluestore = 20 2023-12-25T20:51:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2023-12-25T20:51:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: debug osd = 20 2023-12-25T20:51:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: debug rocksdb = 10 2023-12-25T20:51:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd backfillfull_ratio = 0.85 2023-12-25T20:51:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd full ratio = 0.9 2023-12-25T20:51:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd nearfull ratio = 0.8 2023-12-25T20:51:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify cached snaps = True 2023-12-25T20:51:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify missing on start = True 2023-12-25T20:51:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: osd failsafe full ratio = 0.95 2023-12-25T20:51:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: osd memory target = 1610612736 2023-12-25T20:51:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: osd objectstore = bluestore 2023-12-25T20:51:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: osd op queue = debug_random 2023-12-25T20:51:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: osd op queue cut off = debug_random 2023-12-25T20:51:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.373 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr] 2023-12-25T20:51:52.373 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2023-12-25T20:51:52.373 INFO:teuthology.orchestra.run.smithi153.stdout: debug mgr = 20 2023-12-25T20:51:52.373 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2023-12-25T20:51:52.373 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2023-12-25T20:51:52.373 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2023-12-25T20:51:52.373 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2023-12-25T20:51:52.373 INFO:teuthology.orchestra.run.smithi153.stdout: mgr/telemetry/nag = false 2023-12-25T20:51:52.373 INFO:teuthology.orchestra.run.smithi153.stdout: debug monc = 10 2023-12-25T20:51:52.373 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.373 INFO:teuthology.orchestra.run.smithi153.stdout:[mon] 2023-12-25T20:51:52.373 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2023-12-25T20:51:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2023-12-25T20:51:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: debug paxos = 20 2023-12-25T20:51:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2023-12-25T20:51:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: mon data avail warn = 5 2023-12-25T20:51:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: mon mgr mkfs grace = 240 2023-12-25T20:51:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2023-12-25T20:51:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd reporter subtree level = osd 2023-12-25T20:51:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd prime pg temp = true 2023-12-25T20:51:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2023-12-25T20:51:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-25T20:51:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: auth mon ticket ttl = 660 # 11m 2023-12-25T20:51:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: auth service ticket ttl = 240 # 4m 2023-12-25T20:51:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: # don't complain about insecure global_id in the test suite 2023-12-25T20:51:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-25T20:51:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-25T20:51:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: mon min osdmap epochs = 25 2023-12-25T20:51:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune interval = 2 2023-12-25T20:51:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune min = 15 2023-12-25T20:51:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune txsize = 2 2023-12-25T20:51:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: mon scrub inject crc mismatch = 0.01 2023-12-25T20:51:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: mon scrub inject missing keys = 0.05 2023-12-25T20:51:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: mon scrub interval = 300 2023-12-25T20:51:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: paxos service trim min = 5 2023-12-25T20:51:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:51:52.375 INFO:teuthology.orchestra.run.smithi153.stdout:[client] 2023-12-25T20:51:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: rgw cache enabled = true 2023-12-25T20:51:52.376 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable ops log = true 2023-12-25T20:51:52.376 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable usage log = true 2023-12-25T20:51:52.376 INFO:teuthology.orchestra.run.smithi153.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-25T20:51:52.376 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-25T20:51:52.376 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.a] 2023-12-25T20:51:52.376 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.c] 2023-12-25T20:51:52.376 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.b] 2023-12-25T20:51:52.377 INFO:teuthology.orchestra.run.smithi191.stdout:[global] 2023-12-25T20:51:52.377 INFO:teuthology.orchestra.run.smithi191.stdout: chdir = "" 2023-12-25T20:51:52.377 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-25T20:51:52.377 INFO:teuthology.orchestra.run.smithi191.stdout: auth supported = cephx 2023-12-25T20:51:52.377 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.377 INFO:teuthology.orchestra.run.smithi191.stdout: filestore xattr use omap = true 2023-12-25T20:51:52.377 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.377 INFO:teuthology.orchestra.run.smithi191.stdout: mon clock drift allowed = 1.000 2023-12-25T20:51:52.377 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.377 INFO:teuthology.orchestra.run.smithi191.stdout: osd crush chooseleaf type = 0 2023-12-25T20:51:52.377 INFO:teuthology.orchestra.run.smithi191.stdout: auth debug = true 2023-12-25T20:51:52.377 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.378 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on old message = true 2023-12-25T20:51:52.378 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on bug = true 2023-12-25T20:51:52.378 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.378 INFO:teuthology.orchestra.run.smithi191.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-25T20:51:52.378 INFO:teuthology.orchestra.run.smithi191.stdout: mon pg warn max object skew = 0 2023-12-25T20:51:52.378 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.378 INFO:teuthology.orchestra.run.smithi191.stdout: # disable pg_autoscaler by default for new pools 2023-12-25T20:51:52.378 INFO:teuthology.orchestra.run.smithi191.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-25T20:51:52.378 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.378 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default size = 2 2023-12-25T20:51:52.378 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.378 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow primary affinity = true 2023-12-25T20:51:52.378 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow pg remap = true 2023-12-25T20:51:52.379 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on legacy crush tunables = false 2023-12-25T20:51:52.379 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on crush straw calc version zero = false 2023-12-25T20:51:52.379 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on no sortbitwise = false 2023-12-25T20:51:52.379 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on osd down out interval zero = false 2023-12-25T20:51:52.379 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on too few osds = false 2023-12-25T20:51:52.379 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-25T20:51:52.379 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-25T20:51:52.379 INFO:teuthology.orchestra.run.smithi191.stdout: mon_allow_pool_size_one = true 2023-12-25T20:51:52.379 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.379 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-12-25T20:51:52.379 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.379 INFO:teuthology.orchestra.run.smithi191.stdout: osd default data pool replay window = 5 2023-12-25T20:51:52.379 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.379 INFO:teuthology.orchestra.run.smithi191.stdout: mon allow pool delete = true 2023-12-25T20:51:52.379 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.380 INFO:teuthology.orchestra.run.smithi191.stdout: mon cluster log file level = debug 2023-12-25T20:51:52.380 INFO:teuthology.orchestra.run.smithi191.stdout: debug asserts on shutdown = true 2023-12-25T20:51:52.380 INFO:teuthology.orchestra.run.smithi191.stdout: mon health detail to clog = false 2023-12-25T20:51:52.380 INFO:teuthology.orchestra.run.smithi191.stdout: mon host = "172.21.15.153,[v2:172.21.15.153:3301,v1:172.21.15.153:6790],172.21.15.191" 2023-12-25T20:51:52.380 INFO:teuthology.orchestra.run.smithi191.stdout: mon client directed command retry = 5 2023-12-25T20:51:52.380 INFO:teuthology.orchestra.run.smithi191.stdout: mon election default strategy = 1 2023-12-25T20:51:52.380 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject delay max = 1 2023-12-25T20:51:52.380 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject delay probability = 0.005 2023-12-25T20:51:52.380 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject delay type = mon 2023-12-25T20:51:52.380 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject internal delays = 0.002 2023-12-25T20:51:52.380 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject socket failures = 2500 2023-12-25T20:51:52.380 INFO:teuthology.orchestra.run.smithi191.stdout: ms type = async 2023-12-25T20:51:52.380 INFO:teuthology.orchestra.run.smithi191.stdout: fsid = 0ece9abc-d1cb-4514-a255-e770deeb21b9 2023-12-25T20:51:52.380 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.380 INFO:teuthology.orchestra.run.smithi191.stdout:[osd] 2023-12-25T20:51:52.381 INFO:teuthology.orchestra.run.smithi191.stdout: osd journal size = 100 2023-12-25T20:51:52.381 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.381 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub load threshold = 5.0 2023-12-25T20:51:52.381 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub max interval = 600 2023-12-25T20:51:52.381 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.381 INFO:teuthology.orchestra.run.smithi191.stdout: osd recover clone overlap = true 2023-12-25T20:51:52.381 INFO:teuthology.orchestra.run.smithi191.stdout: osd recovery max chunk = 1048576 2023-12-25T20:51:52.381 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.381 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug shutdown = true 2023-12-25T20:51:52.381 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug op order = true 2023-12-25T20:51:52.381 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify stray on activate = true 2023-12-25T20:51:52.381 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.381 INFO:teuthology.orchestra.run.smithi191.stdout: osd open classes on start = true 2023-12-25T20:51:52.382 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug pg log writeout = true 2023-12-25T20:51:52.382 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.382 INFO:teuthology.orchestra.run.smithi191.stdout: osd deep scrub update digest min age = 30 2023-12-25T20:51:52.382 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.382 INFO:teuthology.orchestra.run.smithi191.stdout: osd map max advance = 10 2023-12-25T20:51:52.382 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.382 INFO:teuthology.orchestra.run.smithi191.stdout: journal zero on create = true 2023-12-25T20:51:52.382 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.382 INFO:teuthology.orchestra.run.smithi191.stdout: filestore ondisk finisher threads = 3 2023-12-25T20:51:52.382 INFO:teuthology.orchestra.run.smithi191.stdout: filestore apply finisher threads = 3 2023-12-25T20:51:52.382 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.382 INFO:teuthology.orchestra.run.smithi191.stdout: bdev debug aio = true 2023-12-25T20:51:52.382 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug misdirected ops = true 2023-12-25T20:51:52.382 INFO:teuthology.orchestra.run.smithi191.stdout: bdev async discard = True 2023-12-25T20:51:52.382 INFO:teuthology.orchestra.run.smithi191.stdout: bdev enable discard = True 2023-12-25T20:51:52.382 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore block size = 96636764160 2023-12-25T20:51:52.383 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore fsck on mount = True 2023-12-25T20:51:52.383 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluefs = 20 2023-12-25T20:51:52.383 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluestore = 20 2023-12-25T20:51:52.383 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2023-12-25T20:51:52.383 INFO:teuthology.orchestra.run.smithi191.stdout: debug osd = 20 2023-12-25T20:51:52.383 INFO:teuthology.orchestra.run.smithi191.stdout: debug rocksdb = 10 2023-12-25T20:51:52.383 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd backfillfull_ratio = 0.85 2023-12-25T20:51:52.383 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd full ratio = 0.9 2023-12-25T20:51:52.383 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd nearfull ratio = 0.8 2023-12-25T20:51:52.383 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify cached snaps = True 2023-12-25T20:51:52.383 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify missing on start = True 2023-12-25T20:51:52.383 INFO:teuthology.orchestra.run.smithi191.stdout: osd failsafe full ratio = 0.95 2023-12-25T20:51:52.383 INFO:teuthology.orchestra.run.smithi191.stdout: osd memory target = 1610612736 2023-12-25T20:51:52.383 INFO:teuthology.orchestra.run.smithi191.stdout: osd objectstore = bluestore 2023-12-25T20:51:52.383 INFO:teuthology.orchestra.run.smithi191.stdout: osd op queue = debug_random 2023-12-25T20:51:52.383 INFO:teuthology.orchestra.run.smithi191.stdout: osd op queue cut off = debug_random 2023-12-25T20:51:52.384 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.384 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr] 2023-12-25T20:51:52.384 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2023-12-25T20:51:52.384 INFO:teuthology.orchestra.run.smithi191.stdout: debug mgr = 20 2023-12-25T20:51:52.384 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2023-12-25T20:51:52.384 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2023-12-25T20:51:52.384 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2023-12-25T20:51:52.384 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2023-12-25T20:51:52.384 INFO:teuthology.orchestra.run.smithi191.stdout: mgr/telemetry/nag = false 2023-12-25T20:51:52.384 INFO:teuthology.orchestra.run.smithi191.stdout: debug monc = 10 2023-12-25T20:51:52.384 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.384 INFO:teuthology.orchestra.run.smithi191.stdout:[mon] 2023-12-25T20:51:52.384 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2023-12-25T20:51:52.384 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2023-12-25T20:51:52.385 INFO:teuthology.orchestra.run.smithi191.stdout: debug paxos = 20 2023-12-25T20:51:52.385 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2023-12-25T20:51:52.385 INFO:teuthology.orchestra.run.smithi191.stdout: mon data avail warn = 5 2023-12-25T20:51:52.385 INFO:teuthology.orchestra.run.smithi191.stdout: mon mgr mkfs grace = 240 2023-12-25T20:51:52.385 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2023-12-25T20:51:52.385 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd reporter subtree level = osd 2023-12-25T20:51:52.385 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd prime pg temp = true 2023-12-25T20:51:52.385 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2023-12-25T20:51:52.385 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.385 INFO:teuthology.orchestra.run.smithi191.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-25T20:51:52.385 INFO:teuthology.orchestra.run.smithi191.stdout: auth mon ticket ttl = 660 # 11m 2023-12-25T20:51:52.385 INFO:teuthology.orchestra.run.smithi191.stdout: auth service ticket ttl = 240 # 4m 2023-12-25T20:51:52.385 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.385 INFO:teuthology.orchestra.run.smithi191.stdout: # don't complain about insecure global_id in the test suite 2023-12-25T20:51:52.386 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-25T20:51:52.386 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-25T20:51:52.386 INFO:teuthology.orchestra.run.smithi191.stdout: mon min osdmap epochs = 25 2023-12-25T20:51:52.386 INFO:teuthology.orchestra.run.smithi191.stdout: mon osdmap full prune interval = 2 2023-12-25T20:51:52.386 INFO:teuthology.orchestra.run.smithi191.stdout: mon osdmap full prune min = 15 2023-12-25T20:51:52.386 INFO:teuthology.orchestra.run.smithi191.stdout: mon osdmap full prune txsize = 2 2023-12-25T20:51:52.386 INFO:teuthology.orchestra.run.smithi191.stdout: mon scrub inject crc mismatch = 0.01 2023-12-25T20:51:52.386 INFO:teuthology.orchestra.run.smithi191.stdout: mon scrub inject missing keys = 0.05 2023-12-25T20:51:52.386 INFO:teuthology.orchestra.run.smithi191.stdout: mon scrub interval = 300 2023-12-25T20:51:52.386 INFO:teuthology.orchestra.run.smithi191.stdout: paxos service trim min = 5 2023-12-25T20:51:52.386 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-25T20:51:52.386 INFO:teuthology.orchestra.run.smithi191.stdout:[client] 2023-12-25T20:51:52.386 INFO:teuthology.orchestra.run.smithi191.stdout: rgw cache enabled = true 2023-12-25T20:51:52.386 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable ops log = true 2023-12-25T20:51:52.386 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable usage log = true 2023-12-25T20:51:52.386 INFO:teuthology.orchestra.run.smithi191.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-25T20:51:52.387 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-25T20:51:52.387 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.a] 2023-12-25T20:51:52.387 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.c] 2023-12-25T20:51:52.387 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.b] 2023-12-25T20:51:52.388 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-25T20:51:52.388 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-25T20:51:52.480 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-25T20:51:52.480 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-12-25T20:51:52.480 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-25T20:51:52.497 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-12-25T20:51:52.497 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-25T20:51:52.554 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi153.front.sepia.ceph.com 2023-12-25T20:51:52.555 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-12-25T20:51:52.555 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-25T20:51:52.555 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-25T20:51:52.634 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-12-25T20:51:52.635 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-25T20:51:52.693 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi191.front.sepia.ceph.com 2023-12-25T20:51:52.693 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-25T20:51:52.693 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-25T20:51:52.694 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-25T20:51:52.732 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-25T20:51:52.732 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-25T20:51:52.791 INFO:tasks.ceph:Setting up mon nodes... 2023-12-25T20:51:52.791 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-25T20:51:52.791 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-25T20:51:52.951 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-25T20:51:52.954 INFO:tasks.ceph:Setting up mds nodes... 2023-12-25T20:51:52.954 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-25T20:51:52.954 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-25T20:51:52.993 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-25T20:51:53.005 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-25T20:51:53.005 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2023-12-25T20:51:53.005 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-25T20:51:53.032 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-25T20:51:53.032 INFO:tasks.ceph:role: osd.0 2023-12-25T20:51:53.032 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com 2023-12-25T20:51:53.032 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-25T20:51:53.107 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-25T20:51:53.107 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-25T20:51:53.108 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-25T20:51:53.108 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-25T20:51:53.108 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-25T20:51:53.108 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2023-12-25T20:51:53.108 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-25T20:51:53.108 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-25T20:51:53.108 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-25T20:51:53.108 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-25T20:51:53.548 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2023-12-25T20:51:53.550 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2023-12-25T20:51:53.550 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-25T20:51:53.738 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-25T20:51:53.772 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-25T20:51:53.840 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-25T20:51:53.840 INFO:tasks.ceph:role: osd.1 2023-12-25T20:51:53.840 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com 2023-12-25T20:51:53.840 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-25T20:51:53.915 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-25T20:51:53.915 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-25T20:51:53.915 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-25T20:51:53.915 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-25T20:51:53.915 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-25T20:51:53.915 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2023-12-25T20:51:53.915 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-25T20:51:53.915 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-25T20:51:53.915 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-25T20:51:53.916 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-25T20:51:54.352 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2023-12-25T20:51:54.353 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2023-12-25T20:51:54.354 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-25T20:51:54.464 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-25T20:51:54.495 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-25T20:51:54.562 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-25T20:51:54.563 INFO:tasks.ceph:role: osd.2 2023-12-25T20:51:54.563 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com 2023-12-25T20:51:54.563 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-25T20:51:54.639 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-25T20:51:54.639 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-25T20:51:54.639 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-25T20:51:54.639 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-25T20:51:54.639 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-25T20:51:54.639 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2023-12-25T20:51:54.639 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-25T20:51:54.639 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-25T20:51:54.639 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-25T20:51:54.639 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-25T20:51:55.079 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2023-12-25T20:51:55.081 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2023-12-25T20:51:55.081 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-25T20:51:55.236 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-25T20:51:55.268 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-25T20:51:55.364 INFO:teuthology.orchestra.run.smithi153.stderr:2023-12-25T20:51:55.363+0000 7f9a216fb380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-12-25T20:51:55.365 INFO:teuthology.orchestra.run.smithi153.stderr:2023-12-25T20:51:55.365+0000 7f9a216fb380 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-12-25T20:51:55.365 INFO:teuthology.orchestra.run.smithi153.stderr:2023-12-25T20:51:55.365+0000 7f9a216fb380 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-25T20:51:55.366 INFO:teuthology.orchestra.run.smithi153.stderr:2023-12-25T20:51:55.365+0000 7f9a216fb380 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-25T20:51:55.366 INFO:teuthology.orchestra.run.smithi153.stderr:2023-12-25T20:51:55.365+0000 7f9a216fb380 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-25T20:51:55.372 INFO:teuthology.orchestra.run.smithi153.stderr:2023-12-25T20:51:55.372+0000 7f9a216fb380 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-12-25T20:51:58.223 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-25T20:51:58.255 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-25T20:51:58.343 INFO:teuthology.orchestra.run.smithi153.stderr:2023-12-25T20:51:58.342+0000 7f3ff626b380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-12-25T20:51:58.343 INFO:teuthology.orchestra.run.smithi153.stderr:2023-12-25T20:51:58.342+0000 7f3ff626b380 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-12-25T20:51:58.343 INFO:teuthology.orchestra.run.smithi153.stderr:2023-12-25T20:51:58.342+0000 7f3ff626b380 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-25T20:51:58.343 INFO:teuthology.orchestra.run.smithi153.stderr:2023-12-25T20:51:58.342+0000 7f3ff626b380 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-25T20:51:58.344 INFO:teuthology.orchestra.run.smithi153.stderr:2023-12-25T20:51:58.342+0000 7f3ff626b380 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-25T20:51:58.344 INFO:teuthology.orchestra.run.smithi153.stderr:2023-12-25T20:51:58.343+0000 7f3ff626b380 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-12-25T20:52:01.183 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-25T20:52:01.214 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-25T20:52:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2023-12-25T20:52:01.302+0000 7fa1c44de380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-12-25T20:52:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2023-12-25T20:52:01.302+0000 7fa1c44de380 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-12-25T20:52:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2023-12-25T20:52:01.302+0000 7fa1c44de380 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-25T20:52:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2023-12-25T20:52:01.302+0000 7fa1c44de380 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-25T20:52:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2023-12-25T20:52:01.302+0000 7fa1c44de380 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-25T20:52:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:2023-12-25T20:52:01.303+0000 7fa1c44de380 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-12-25T20:52:04.143 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-25T20:52:04.175 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-12-25T20:52:04.209 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-12-25T20:52:04.210 INFO:tasks.ceph:role: osd.3 2023-12-25T20:52:04.210 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com 2023-12-25T20:52:04.210 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-25T20:52:04.288 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-25T20:52:04.288 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-25T20:52:04.288 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-25T20:52:04.288 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-25T20:52:04.288 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-25T20:52:04.288 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2023-12-25T20:52:04.288 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-25T20:52:04.288 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-25T20:52:04.289 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-25T20:52:04.289 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-25T20:52:04.721 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2023-12-25T20:52:04.723 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2023-12-25T20:52:04.723 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2023-12-25T20:52:04.895 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-12-25T20:52:04.934 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-12-25T20:52:05.004 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-12-25T20:52:05.004 INFO:tasks.ceph:role: osd.4 2023-12-25T20:52:05.004 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com 2023-12-25T20:52:05.004 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-25T20:52:05.081 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-25T20:52:05.081 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-25T20:52:05.081 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-25T20:52:05.081 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-25T20:52:05.081 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-25T20:52:05.081 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2023-12-25T20:52:05.081 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-25T20:52:05.081 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-25T20:52:05.081 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-25T20:52:05.081 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-25T20:52:05.504 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2023-12-25T20:52:05.506 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2023-12-25T20:52:05.506 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2023-12-25T20:52:05.620 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-12-25T20:52:05.654 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-12-25T20:52:05.723 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-12-25T20:52:05.723 INFO:tasks.ceph:role: osd.5 2023-12-25T20:52:05.723 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com 2023-12-25T20:52:05.724 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-25T20:52:05.801 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-25T20:52:05.801 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-25T20:52:05.801 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-25T20:52:05.801 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-25T20:52:05.801 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-25T20:52:05.801 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2023-12-25T20:52:05.801 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-25T20:52:05.801 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-25T20:52:05.802 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-25T20:52:05.802 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-25T20:52:06.244 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2023-12-25T20:52:06.246 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2023-12-25T20:52:06.246 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2023-12-25T20:52:06.398 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-12-25T20:52:06.433 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-25T20:52:06.530 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-25T20:52:06.530+0000 7fd3e523f380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-12-25T20:52:06.531 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-25T20:52:06.530+0000 7fd3e523f380 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-12-25T20:52:06.531 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-25T20:52:06.531+0000 7fd3e523f380 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-25T20:52:06.531 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-25T20:52:06.531+0000 7fd3e523f380 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-25T20:52:06.532 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-25T20:52:06.531+0000 7fd3e523f380 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-25T20:52:06.538 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-25T20:52:06.538+0000 7fd3e523f380 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-12-25T20:52:09.392 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-12-25T20:52:09.425 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-25T20:52:09.514 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-25T20:52:09.513+0000 7ffbba838380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-12-25T20:52:09.514 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-25T20:52:09.513+0000 7ffbba838380 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-12-25T20:52:09.514 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-25T20:52:09.513+0000 7ffbba838380 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-25T20:52:09.514 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-25T20:52:09.513+0000 7ffbba838380 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-25T20:52:09.514 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-25T20:52:09.513+0000 7ffbba838380 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-25T20:52:09.515 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-25T20:52:09.514+0000 7ffbba838380 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-12-25T20:52:12.362 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-12-25T20:52:12.394 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-25T20:52:12.482 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-25T20:52:12.481+0000 7f8ec4758380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-12-25T20:52:12.482 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-25T20:52:12.481+0000 7f8ec4758380 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-12-25T20:52:12.482 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-25T20:52:12.481+0000 7f8ec4758380 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-25T20:52:12.482 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-25T20:52:12.481+0000 7f8ec4758380 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-25T20:52:12.482 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-25T20:52:12.481+0000 7f8ec4758380 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-25T20:52:12.483 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-25T20:52:12.482+0000 7f8ec4758380 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-12-25T20:52:15.324 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-12-25T20:52:15.354 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-25T20:52:15.354 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-12-25T20:52:15.354 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-25T20:52:15.387 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-12-25T20:52:15.387 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-25T20:52:15.460 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-12-25T20:52:15.460 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-25T20:52:15.531 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-25T20:52:15.531 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-25T20:52:15.562 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-25T20:52:15.563 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-12-25T20:52:15.632 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-25T20:52:15.633 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-12-25T20:52:15.701 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-25T20:52:15.701 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-12-25T20:52:15.770 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-25T20:52:15.770 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-25T20:52:15.831 INFO:tasks.ceph:Adding keys to all mons... 2023-12-25T20:52:15.831 DEBUG:teuthology.orchestra.run.smithi153:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-25T20:52:15.834 DEBUG:teuthology.orchestra.run.smithi191:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-25T20:52:15.875 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.0] 2023-12-25T20:52:15.875 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBr64ll4oe/FRAAdKpyZX7vVpc5E/DANZv8xA== 2023-12-25T20:52:15.875 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.1] 2023-12-25T20:52:15.875 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBu64llk3l/FBAAvZmIRUTkfI5up/MYI2AORQ== 2023-12-25T20:52:15.875 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.2] 2023-12-25T20:52:15.875 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBx64llS1YWEhAAjltrMi1jJ5kKl7dQKvxsbA== 2023-12-25T20:52:15.875 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr.x] 2023-12-25T20:52:15.875 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBo64llSTXIOBAARX2J4P4zwHjEj7LdHRMxZA== 2023-12-25T20:52:15.875 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.3] 2023-12-25T20:52:15.875 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQB264llBcGvHxAAsIsOJWzLnpeEz6CKee/Wgw== 2023-12-25T20:52:15.875 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.4] 2023-12-25T20:52:15.876 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQB564llomGwHhAAUt/4fQjFp09M0e5vN83lAw== 2023-12-25T20:52:15.876 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.5] 2023-12-25T20:52:15.876 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQB864llCyTBHBAAygyhWXrs0Lv3hpdpEmx/tg== 2023-12-25T20:52:15.876 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2023-12-25T20:52:15.876 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBo64llr2RBOxAAPZJDJqLrt4y4nhlvuhWGTA== 2023-12-25T20:52:15.898 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.0] 2023-12-25T20:52:15.899 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQBr64ll4oe/FRAAdKpyZX7vVpc5E/DANZv8xA== 2023-12-25T20:52:15.899 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.1] 2023-12-25T20:52:15.899 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQBu64llk3l/FBAAvZmIRUTkfI5up/MYI2AORQ== 2023-12-25T20:52:15.899 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.2] 2023-12-25T20:52:15.899 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQBx64llS1YWEhAAjltrMi1jJ5kKl7dQKvxsbA== 2023-12-25T20:52:15.899 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr.x] 2023-12-25T20:52:15.899 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQBo64llSTXIOBAARX2J4P4zwHjEj7LdHRMxZA== 2023-12-25T20:52:15.899 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.3] 2023-12-25T20:52:15.899 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQB264llBcGvHxAAsIsOJWzLnpeEz6CKee/Wgw== 2023-12-25T20:52:15.899 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.4] 2023-12-25T20:52:15.899 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQB564llomGwHhAAUt/4fQjFp09M0e5vN83lAw== 2023-12-25T20:52:15.899 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.5] 2023-12-25T20:52:15.899 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQB864llCyTBHBAAygyhWXrs0Lv3hpdpEmx/tg== 2023-12-25T20:52:15.899 INFO:teuthology.orchestra.run.smithi191.stdout:[client.0] 2023-12-25T20:52:15.900 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQBo64llr2RBOxAAPZJDJqLrt4y4nhlvuhWGTA== 2023-12-25T20:52:15.900 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-25T20:52:15.920 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-25T20:52:15.991 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-25T20:52:15.993 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-25T20:52:16.088 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-25T20:52:16.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-25T20:52:16.140 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-25T20:52:16.142 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-25T20:52:16.190 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-25T20:52:16.192 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-25T20:52:16.240 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-25T20:52:16.243 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-25T20:52:16.291 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-25T20:52:16.293 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-25T20:52:16.341 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-25T20:52:16.344 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-25T20:52:16.394 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-25T20:52:16.394 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-25T20:52:16.421 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-25T20:52:16.514 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-25T20:52:16.541 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-12-25T20:52:16.609 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-25T20:52:16.701 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-12-25T20:52:16.728 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-12-25T20:52:16.757 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-25T20:52:16.851 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-12-25T20:52:16.877 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-25T20:52:16.880 DEBUG:teuthology.orchestra.run.smithi191:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-25T20:52:16.935 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-25T20:52:16.935 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-25T20:52:16.936 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-12-25T20:52:16.938 INFO:tasks.ceph.mon.a:Started 2023-12-25T20:52:16.938 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-25T20:52:16.938 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-12-25T20:52:16.940 INFO:tasks.ceph.mon.c:Started 2023-12-25T20:52:16.940 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-25T20:52:16.940 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-12-25T20:52:16.978 INFO:tasks.ceph.mon.b:Started 2023-12-25T20:52:16.978 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-25T20:52:16.978 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-25T20:52:16.978 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-12-25T20:52:16.980 INFO:tasks.ceph.mgr.x:Started 2023-12-25T20:52:16.981 INFO:tasks.ceph:Setting crush tunables to default 2023-12-25T20:52:16.981 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd crush tunables default 2023-12-25T20:52:22.806 INFO:teuthology.orchestra.run.smithi153.stderr:adjusted tunables profile to default 2023-12-25T20:52:22.819 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-25T20:52:22.820 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-12-25T20:52:22.820 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-25T20:52:22.849 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-12-25T20:52:22.850 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-25T20:52:22.919 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-12-25T20:52:22.919 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-25T20:52:22.988 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-25T20:52:22.988 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-12-25T20:52:23.021 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-25T20:52:23.021 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-12-25T20:52:23.091 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-25T20:52:23.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-12-25T20:52:23.159 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 595d263d-6a56-4ccc-bb15-cd8d72e49f45 0 2023-12-25T20:52:23.205 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:23.206+0000 7f86dca76680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-25T20:52:23.278 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:23.278+0000 7f86dca76680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-25T20:52:23.420 INFO:teuthology.orchestra.run.smithi191.stdout:0 2023-12-25T20:52:23.433 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 3b8105a7-7f13-49a6-9388-23a4d9d131e1 1 2023-12-25T20:52:23.683 INFO:teuthology.orchestra.run.smithi191.stdout:1 2023-12-25T20:52:23.696 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 6b62f25d-a6fa-4089-9fd1-1b08d2cbad57 2 2023-12-25T20:52:23.729 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:23.729+0000 7f86dca76680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-25T20:52:23.907 INFO:teuthology.orchestra.run.smithi191.stdout:2 2023-12-25T20:52:23.919 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 751e8edf-7ff9-4b73-9e8d-38416db248a1 3 2023-12-25T20:52:24.168 INFO:teuthology.orchestra.run.smithi191.stdout:3 2023-12-25T20:52:24.180 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 7635bbb7-ef87-483c-aadd-1984f330944c 4 2023-12-25T20:52:24.392 INFO:teuthology.orchestra.run.smithi191.stdout:4 2023-12-25T20:52:24.406 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 7519398e-167d-40be-847b-1bea65ef0ea4 5 2023-12-25T20:52:24.482 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:24.482+0000 7f86dca76680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-25T20:52:24.551 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:24.552+0000 7f86dca76680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-25T20:52:24.638 INFO:teuthology.orchestra.run.smithi191.stdout:5 2023-12-25T20:52:24.650 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-25T20:52:24.650 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-12-25T20:52:24.653 INFO:tasks.ceph.osd.0:Started 2023-12-25T20:52:24.653 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-25T20:52:24.653 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-12-25T20:52:24.655 INFO:tasks.ceph.osd.1:Started 2023-12-25T20:52:24.655 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-25T20:52:24.656 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-12-25T20:52:24.658 INFO:tasks.ceph.osd.2:Started 2023-12-25T20:52:24.659 INFO:tasks.ceph.osd.3:Restarting daemon 2023-12-25T20:52:24.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-12-25T20:52:24.662 INFO:tasks.ceph.osd.3:Started 2023-12-25T20:52:24.662 INFO:tasks.ceph.osd.4:Restarting daemon 2023-12-25T20:52:24.662 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-12-25T20:52:24.665 INFO:tasks.ceph.osd.4:Started 2023-12-25T20:52:24.665 INFO:tasks.ceph.osd.5:Restarting daemon 2023-12-25T20:52:24.665 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-12-25T20:52:24.667 INFO:tasks.ceph.osd.5:Started 2023-12-25T20:52:24.668 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-25T20:52:24.757 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:24.758+0000 7f86dca76680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-25T20:52:24.877 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:52:24.877 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":8,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","created":"2023-12-25T20:52:22.419978+0000","modified":"2023-12-25T20:52:24.636112+0000","last_up_change":"0.000000","last_in_change":"2023-12-25T20:52:24.636112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"595d263d-6a56-4ccc-bb15-cd8d72e49f45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3b8105a7-7f13-49a6-9388-23a4d9d131e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6b62f25d-a6fa-4089-9fd1-1b08d2cbad57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"751e8edf-7ff9-4b73-9e8d-38416db248a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7635bbb7-ef87-483c-aadd-1984f330944c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7519398e-167d-40be-847b-1bea65ef0ea4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-25T20:52:24.888 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-25T20:52:24.888 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-25T20:52:25.196 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:25.196+0000 7f86dca76680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-25T20:52:25.257 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-12-25T20:52:25.257+0000 7fea3dd29380 -1 Falling back to public interface 2023-12-25T20:52:25.263 INFO:tasks.ceph.osd.3.smithi191.stderr:2023-12-25T20:52:25.264+0000 7f3456e6f380 -1 Falling back to public interface 2023-12-25T20:52:25.267 INFO:tasks.ceph.osd.1.smithi153.stderr:2023-12-25T20:52:25.267+0000 7f4873362380 -1 Falling back to public interface 2023-12-25T20:52:25.268 INFO:tasks.ceph.osd.4.smithi191.stderr:2023-12-25T20:52:25.268+0000 7ff401078380 -1 Falling back to public interface 2023-12-25T20:52:25.271 INFO:tasks.ceph.osd.5.smithi191.stderr:2023-12-25T20:52:25.271+0000 7f3306b07380 -1 Falling back to public interface 2023-12-25T20:52:25.274 INFO:tasks.ceph.osd.2.smithi153.stderr:2023-12-25T20:52:25.274+0000 7f5297df5380 -1 Falling back to public interface 2023-12-25T20:52:25.304 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:25.304+0000 7f86dca76680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-25T20:52:25.457 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:25.457+0000 7f86dca76680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-25T20:52:25.493 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-25T20:52:25.533 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:25.533+0000 7f86dca76680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-25T20:52:25.604 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:25.605+0000 7f86dca76680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-25T20:52:25.680 INFO:teuthology.misc.health.smithi153.stdout: 2023-12-25T20:52:25.680 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":8,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","created":"2023-12-25T20:52:22.419978+0000","modified":"2023-12-25T20:52:24.636112+0000","last_up_change":"0.000000","last_in_change":"2023-12-25T20:52:24.636112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"595d263d-6a56-4ccc-bb15-cd8d72e49f45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3b8105a7-7f13-49a6-9388-23a4d9d131e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6b62f25d-a6fa-4089-9fd1-1b08d2cbad57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"751e8edf-7ff9-4b73-9e8d-38416db248a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7635bbb7-ef87-483c-aadd-1984f330944c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7519398e-167d-40be-847b-1bea65ef0ea4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-25T20:52:25.691 DEBUG:teuthology.misc:0 of 6 OSDs are up 2023-12-25T20:52:25.863 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:25.864+0000 7f86dca76680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-25T20:52:26.060 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:26.060+0000 7f86dca76680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-25T20:52:26.726 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:26.726+0000 7f86dca76680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-25T20:52:26.794 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:26.795+0000 7f86dca76680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-25T20:52:26.873 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:26.874+0000 7f86dca76680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-25T20:52:27.022 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:27.022+0000 7f86dca76680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-25T20:52:27.094 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:27.093+0000 7f86dca76680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-25T20:52:27.211 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:27.211+0000 7f86dca76680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-25T20:52:27.318 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:27.318+0000 7f86dca76680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-25T20:52:27.641 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:27.641+0000 7f86dca76680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-25T20:52:27.714 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:27.714+0000 7f86dca76680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-25T20:52:28.701 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-12-25T20:52:28.701+0000 7fea3dd29380 -1 osd.0 0 log_to_monitors {default=true} 2023-12-25T20:52:28.715 INFO:tasks.ceph.osd.3.smithi191.stderr:2023-12-25T20:52:28.716+0000 7f3456e6f380 -1 osd.3 0 log_to_monitors {default=true} 2023-12-25T20:52:28.741 INFO:tasks.ceph.osd.1.smithi153.stderr:2023-12-25T20:52:28.742+0000 7f4873362380 -1 osd.1 0 log_to_monitors {default=true} 2023-12-25T20:52:28.742 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:28.742+0000 7f8691f8f700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-12-25T20:52:28.742 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-25T20:52:28.742+0000 7f8691f8f700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-12-25T20:52:28.771 INFO:tasks.ceph.osd.2.smithi153.stderr:2023-12-25T20:52:28.772+0000 7f5297df5380 -1 osd.2 0 log_to_monitors {default=true} 2023-12-25T20:52:28.790 INFO:tasks.ceph.osd.5.smithi191.stderr:2023-12-25T20:52:28.790+0000 7f3306b07380 -1 osd.5 0 log_to_monitors {default=true} 2023-12-25T20:52:28.791 INFO:tasks.ceph.osd.4.smithi191.stderr:2023-12-25T20:52:28.792+0000 7ff401078380 -1 osd.4 0 log_to_monitors {default=true} 2023-12-25T20:52:30.261 INFO:tasks.ceph.osd.5.smithi191.stderr:2023-12-25T20:52:30.260+0000 7f32fd4a8700 -1 osd.5 0 waiting for initial osdmap 2023-12-25T20:52:30.261 INFO:tasks.ceph.osd.4.smithi191.stderr:2023-12-25T20:52:30.261+0000 7ff3f7a19700 -1 osd.4 0 waiting for initial osdmap 2023-12-25T20:52:30.263 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-12-25T20:52:30.264+0000 7fea346ca700 -1 osd.0 0 waiting for initial osdmap 2023-12-25T20:52:30.263 INFO:tasks.ceph.osd.1.smithi153.stderr:2023-12-25T20:52:30.264+0000 7f4869d03700 -1 osd.1 0 waiting for initial osdmap 2023-12-25T20:52:30.265 INFO:tasks.ceph.osd.3.smithi191.stderr:2023-12-25T20:52:30.265+0000 7f344d810700 -1 osd.3 0 waiting for initial osdmap 2023-12-25T20:52:30.267 INFO:tasks.ceph.osd.5.smithi191.stderr:2023-12-25T20:52:30.267+0000 7f32f7c20700 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-25T20:52:30.267 INFO:tasks.ceph.osd.4.smithi191.stderr:2023-12-25T20:52:30.267+0000 7ff3f2191700 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-25T20:52:30.269 INFO:tasks.ceph.osd.1.smithi153.stderr:2023-12-25T20:52:30.269+0000 7f486447b700 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-25T20:52:30.270 INFO:tasks.ceph.osd.3.smithi191.stderr:2023-12-25T20:52:30.269+0000 7f3447f88700 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-25T20:52:30.270 INFO:tasks.ceph.osd.2.smithi153.stderr:2023-12-25T20:52:30.271+0000 7f528e796700 -1 osd.2 0 waiting for initial osdmap 2023-12-25T20:52:30.273 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-12-25T20:52:30.273+0000 7fea2ee42700 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-25T20:52:30.275 INFO:tasks.ceph.osd.2.smithi153.stderr:2023-12-25T20:52:30.276+0000 7f5288f0e700 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-25T20:52:30.289 INFO:tasks.ceph.osd.3.smithi191.stderr:2023-12-25T20:52:30.288+0000 7f343d6bc700 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-25T20:52:32.297 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-25T20:52:32.587 INFO:teuthology.misc.health.smithi153.stdout: 2023-12-25T20:52:32.588 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":12,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","created":"2023-12-25T20:52:22.419978+0000","modified":"2023-12-25T20:52:32.273733+0000","last_up_change":"2023-12-25T20:52:31.268017+0000","last_in_change":"2023-12-25T20:52:24.636112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-25T20:52:31.304424+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"595d263d-6a56-4ccc-bb15-cd8d72e49f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":114394},{"type":"v1","addr":"172.21.15.153:6801","nonce":114394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":114394},{"type":"v1","addr":"172.21.15.153:6803","nonce":114394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":114394},{"type":"v1","addr":"172.21.15.153:6807","nonce":114394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":114394},{"type":"v1","addr":"172.21.15.153:6805","nonce":114394}]},"public_addr":"172.21.15.153:6801/114394","cluster_addr":"172.21.15.153:6803/114394","heartbeat_back_addr":"172.21.15.153:6807/114394","heartbeat_front_addr":"172.21.15.153:6805/114394","state":["exists","up"]},{"osd":1,"uuid":"3b8105a7-7f13-49a6-9388-23a4d9d131e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":114395},{"type":"v1","addr":"172.21.15.153:6809","nonce":114395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":114395},{"type":"v1","addr":"172.21.15.153:6811","nonce":114395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":114395},{"type":"v1","addr":"172.21.15.153:6815","nonce":114395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":114395},{"type":"v1","addr":"172.21.15.153:6813","nonce":114395}]},"public_addr":"172.21.15.153:6809/114395","cluster_addr":"172.21.15.153:6811/114395","heartbeat_back_addr":"172.21.15.153:6815/114395","heartbeat_front_addr":"172.21.15.153:6813/114395","state":["exists","up"]},{"osd":2,"uuid":"6b62f25d-a6fa-4089-9fd1-1b08d2cbad57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":114396},{"type":"v1","addr":"172.21.15.153:6817","nonce":114396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":114396},{"type":"v1","addr":"172.21.15.153:6819","nonce":114396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":114396},{"type":"v1","addr":"172.21.15.153:6823","nonce":114396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":114396},{"type":"v1","addr":"172.21.15.153:6821","nonce":114396}]},"public_addr":"172.21.15.153:6817/114396","cluster_addr":"172.21.15.153:6819/114396","heartbeat_back_addr":"172.21.15.153:6823/114396","heartbeat_front_addr":"172.21.15.153:6821/114396","state":["exists","up"]},{"osd":3,"uuid":"751e8edf-7ff9-4b73-9e8d-38416db248a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":114428},{"type":"v1","addr":"172.21.15.191:6801","nonce":114428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":114428},{"type":"v1","addr":"172.21.15.191:6803","nonce":114428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":114428},{"type":"v1","addr":"172.21.15.191:6807","nonce":114428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":114428},{"type":"v1","addr":"172.21.15.191:6805","nonce":114428}]},"public_addr":"172.21.15.191:6801/114428","cluster_addr":"172.21.15.191:6803/114428","heartbeat_back_addr":"172.21.15.191:6807/114428","heartbeat_front_addr":"172.21.15.191:6805/114428","state":["exists","up"]},{"osd":4,"uuid":"7635bbb7-ef87-483c-aadd-1984f330944c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":114429},{"type":"v1","addr":"172.21.15.191:6809","nonce":114429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":114429},{"type":"v1","addr":"172.21.15.191:6811","nonce":114429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":114429},{"type":"v1","addr":"172.21.15.191:6815","nonce":114429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":114429},{"type":"v1","addr":"172.21.15.191:6813","nonce":114429}]},"public_addr":"172.21.15.191:6809/114429","cluster_addr":"172.21.15.191:6811/114429","heartbeat_back_addr":"172.21.15.191:6815/114429","heartbeat_front_addr":"172.21.15.191:6813/114429","state":["exists","up"]},{"osd":5,"uuid":"7519398e-167d-40be-847b-1bea65ef0ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":114430},{"type":"v1","addr":"172.21.15.191:6817","nonce":114430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":114430},{"type":"v1","addr":"172.21.15.191:6819","nonce":114430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":114430},{"type":"v1","addr":"172.21.15.191:6823","nonce":114430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":114430},{"type":"v1","addr":"172.21.15.191:6821","nonce":114430}]},"public_addr":"172.21.15.191:6817/114430","cluster_addr":"172.21.15.191:6819/114430","heartbeat_back_addr":"172.21.15.191:6823/114430","heartbeat_front_addr":"172.21.15.191:6821/114430","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.682269+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.746139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.820727+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.689025+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.837599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.746784+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-25T20:52:32.600 DEBUG:teuthology.misc:6 of 6 OSDs are up 2023-12-25T20:52:32.600 INFO:tasks.ceph:Creating RBD pool 2023-12-25T20:52:32.600 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd pool create rbd 8 2023-12-25T20:52:33.291 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'rbd' created 2023-12-25T20:52:33.304 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-25T20:52:34.297 INFO:teuthology.orchestra.run.smithi153.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-25T20:52:34.310 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-25T20:52:34.310 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-25T20:52:34.310 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-25T20:52:34.311 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-25T20:52:34.311 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-25T20:52:34.652 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:52:34.664 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":4,"active_gid":4108,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":113916},{"type":"v1","addr":"172.21.15.191:6825","nonce":113916}]},"active_addr":"172.21.15.191:6825/113916","active_change":"2023-12-25T20:52:27.727754+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.191:0","nonce":2916250222}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.191:0","nonce":1444836103}]}]} 2023-12-25T20:52:34.667 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-25T20:52:34.667 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-25T20:52:34.668 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-25T20:52:34.966 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:52:34.967 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":14,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","created":"2023-12-25T20:52:22.419978+0000","modified":"2023-12-25T20:52:34.285612+0000","last_up_change":"2023-12-25T20:52:31.268017+0000","last_in_change":"2023-12-25T20:52:24.636112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-25T20:52:31.304424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-25T20:52:32.915209+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"595d263d-6a56-4ccc-bb15-cd8d72e49f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":114394},{"type":"v1","addr":"172.21.15.153:6801","nonce":114394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":114394},{"type":"v1","addr":"172.21.15.153:6803","nonce":114394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":114394},{"type":"v1","addr":"172.21.15.153:6807","nonce":114394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":114394},{"type":"v1","addr":"172.21.15.153:6805","nonce":114394}]},"public_addr":"172.21.15.153:6801/114394","cluster_addr":"172.21.15.153:6803/114394","heartbeat_back_addr":"172.21.15.153:6807/114394","heartbeat_front_addr":"172.21.15.153:6805/114394","state":["exists","up"]},{"osd":1,"uuid":"3b8105a7-7f13-49a6-9388-23a4d9d131e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":114395},{"type":"v1","addr":"172.21.15.153:6809","nonce":114395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":114395},{"type":"v1","addr":"172.21.15.153:6811","nonce":114395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":114395},{"type":"v1","addr":"172.21.15.153:6815","nonce":114395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":114395},{"type":"v1","addr":"172.21.15.153:6813","nonce":114395}]},"public_addr":"172.21.15.153:6809/114395","cluster_addr":"172.21.15.153:6811/114395","heartbeat_back_addr":"172.21.15.153:6815/114395","heartbeat_front_addr":"172.21.15.153:6813/114395","state":["exists","up"]},{"osd":2,"uuid":"6b62f25d-a6fa-4089-9fd1-1b08d2cbad57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":114396},{"type":"v1","addr":"172.21.15.153:6817","nonce":114396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":114396},{"type":"v1","addr":"172.21.15.153:6819","nonce":114396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":114396},{"type":"v1","addr":"172.21.15.153:6823","nonce":114396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":114396},{"type":"v1","addr":"172.21.15.153:6821","nonce":114396}]},"public_addr":"172.21.15.153:6817/114396","cluster_addr":"172.21.15.153:6819/114396","heartbeat_back_addr":"172.21.15.153:6823/114396","heartbeat_front_addr":"172.21.15.153:6821/114396","state":["exists","up"]},{"osd":3,"uuid":"751e8edf-7ff9-4b73-9e8d-38416db248a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":114428},{"type":"v1","addr":"172.21.15.191:6801","nonce":114428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":114428},{"type":"v1","addr":"172.21.15.191:6803","nonce":114428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":114428},{"type":"v1","addr":"172.21.15.191:6807","nonce":114428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":114428},{"type":"v1","addr":"172.21.15.191:6805","nonce":114428}]},"public_addr":"172.21.15.191:6801/114428","cluster_addr":"172.21.15.191:6803/114428","heartbeat_back_addr":"172.21.15.191:6807/114428","heartbeat_front_addr":"172.21.15.191:6805/114428","state":["exists","up"]},{"osd":4,"uuid":"7635bbb7-ef87-483c-aadd-1984f330944c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":114429},{"type":"v1","addr":"172.21.15.191:6809","nonce":114429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":114429},{"type":"v1","addr":"172.21.15.191:6811","nonce":114429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":114429},{"type":"v1","addr":"172.21.15.191:6815","nonce":114429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":114429},{"type":"v1","addr":"172.21.15.191:6813","nonce":114429}]},"public_addr":"172.21.15.191:6809/114429","cluster_addr":"172.21.15.191:6811/114429","heartbeat_back_addr":"172.21.15.191:6815/114429","heartbeat_front_addr":"172.21.15.191:6813/114429","state":["exists","up"]},{"osd":5,"uuid":"7519398e-167d-40be-847b-1bea65ef0ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":114430},{"type":"v1","addr":"172.21.15.191:6817","nonce":114430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":114430},{"type":"v1","addr":"172.21.15.191:6819","nonce":114430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":114430},{"type":"v1","addr":"172.21.15.191:6823","nonce":114430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":114430},{"type":"v1","addr":"172.21.15.191:6821","nonce":114430}]},"public_addr":"172.21.15.191:6817/114430","cluster_addr":"172.21.15.191:6819/114430","heartbeat_back_addr":"172.21.15.191:6823/114430","heartbeat_front_addr":"172.21.15.191:6821/114430","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.682269+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.746139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.820727+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.689025+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.837599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.746784+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-25T20:52:34.980 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-25T20:52:34.980 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-25T20:52:35.271 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:52:35.271 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":14,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","created":"2023-12-25T20:52:22.419978+0000","modified":"2023-12-25T20:52:34.285612+0000","last_up_change":"2023-12-25T20:52:31.268017+0000","last_in_change":"2023-12-25T20:52:24.636112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-25T20:52:31.304424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-25T20:52:32.915209+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"595d263d-6a56-4ccc-bb15-cd8d72e49f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":114394},{"type":"v1","addr":"172.21.15.153:6801","nonce":114394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":114394},{"type":"v1","addr":"172.21.15.153:6803","nonce":114394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":114394},{"type":"v1","addr":"172.21.15.153:6807","nonce":114394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":114394},{"type":"v1","addr":"172.21.15.153:6805","nonce":114394}]},"public_addr":"172.21.15.153:6801/114394","cluster_addr":"172.21.15.153:6803/114394","heartbeat_back_addr":"172.21.15.153:6807/114394","heartbeat_front_addr":"172.21.15.153:6805/114394","state":["exists","up"]},{"osd":1,"uuid":"3b8105a7-7f13-49a6-9388-23a4d9d131e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":114395},{"type":"v1","addr":"172.21.15.153:6809","nonce":114395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":114395},{"type":"v1","addr":"172.21.15.153:6811","nonce":114395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":114395},{"type":"v1","addr":"172.21.15.153:6815","nonce":114395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":114395},{"type":"v1","addr":"172.21.15.153:6813","nonce":114395}]},"public_addr":"172.21.15.153:6809/114395","cluster_addr":"172.21.15.153:6811/114395","heartbeat_back_addr":"172.21.15.153:6815/114395","heartbeat_front_addr":"172.21.15.153:6813/114395","state":["exists","up"]},{"osd":2,"uuid":"6b62f25d-a6fa-4089-9fd1-1b08d2cbad57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":114396},{"type":"v1","addr":"172.21.15.153:6817","nonce":114396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":114396},{"type":"v1","addr":"172.21.15.153:6819","nonce":114396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":114396},{"type":"v1","addr":"172.21.15.153:6823","nonce":114396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":114396},{"type":"v1","addr":"172.21.15.153:6821","nonce":114396}]},"public_addr":"172.21.15.153:6817/114396","cluster_addr":"172.21.15.153:6819/114396","heartbeat_back_addr":"172.21.15.153:6823/114396","heartbeat_front_addr":"172.21.15.153:6821/114396","state":["exists","up"]},{"osd":3,"uuid":"751e8edf-7ff9-4b73-9e8d-38416db248a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":114428},{"type":"v1","addr":"172.21.15.191:6801","nonce":114428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":114428},{"type":"v1","addr":"172.21.15.191:6803","nonce":114428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":114428},{"type":"v1","addr":"172.21.15.191:6807","nonce":114428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":114428},{"type":"v1","addr":"172.21.15.191:6805","nonce":114428}]},"public_addr":"172.21.15.191:6801/114428","cluster_addr":"172.21.15.191:6803/114428","heartbeat_back_addr":"172.21.15.191:6807/114428","heartbeat_front_addr":"172.21.15.191:6805/114428","state":["exists","up"]},{"osd":4,"uuid":"7635bbb7-ef87-483c-aadd-1984f330944c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":114429},{"type":"v1","addr":"172.21.15.191:6809","nonce":114429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":114429},{"type":"v1","addr":"172.21.15.191:6811","nonce":114429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":114429},{"type":"v1","addr":"172.21.15.191:6815","nonce":114429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":114429},{"type":"v1","addr":"172.21.15.191:6813","nonce":114429}]},"public_addr":"172.21.15.191:6809/114429","cluster_addr":"172.21.15.191:6811/114429","heartbeat_back_addr":"172.21.15.191:6815/114429","heartbeat_front_addr":"172.21.15.191:6813/114429","state":["exists","up"]},{"osd":5,"uuid":"7519398e-167d-40be-847b-1bea65ef0ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":114430},{"type":"v1","addr":"172.21.15.191:6817","nonce":114430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":114430},{"type":"v1","addr":"172.21.15.191:6819","nonce":114430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":114430},{"type":"v1","addr":"172.21.15.191:6823","nonce":114430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":114430},{"type":"v1","addr":"172.21.15.191:6821","nonce":114430}]},"public_addr":"172.21.15.191:6817/114430","cluster_addr":"172.21.15.191:6819/114430","heartbeat_back_addr":"172.21.15.191:6823/114430","heartbeat_front_addr":"172.21.15.191:6821/114430","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.682269+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.746139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.820727+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.689025+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.837599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.746784+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-25T20:52:35.284 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-25T20:52:35.441 INFO:teuthology.orchestra.run.smithi153.stdout:47244640259 2023-12-25T20:52:35.441 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-25T20:52:35.596 INFO:teuthology.orchestra.run.smithi153.stdout:47244640259 2023-12-25T20:52:35.596 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-25T20:52:35.751 INFO:teuthology.orchestra.run.smithi153.stdout:47244640260 2023-12-25T20:52:35.752 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-12-25T20:52:35.908 INFO:teuthology.orchestra.run.smithi153.stdout:47244640259 2023-12-25T20:52:35.909 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-12-25T20:52:36.065 INFO:teuthology.orchestra.run.smithi153.stdout:47244640259 2023-12-25T20:52:36.065 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-12-25T20:52:36.224 INFO:teuthology.orchestra.run.smithi153.stdout:47244640259 2023-12-25T20:52:36.224 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-25T20:52:36.514 INFO:teuthology.orchestra.run.smithi153.stdout:47244640259 2023-12-25T20:52:36.527 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.0 2023-12-25T20:52:36.528 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-25T20:52:36.819 INFO:teuthology.orchestra.run.smithi153.stdout:47244640259 2023-12-25T20:52:36.833 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.1 2023-12-25T20:52:36.833 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-25T20:52:37.126 INFO:teuthology.orchestra.run.smithi153.stdout:47244640259 2023-12-25T20:52:37.139 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.2 2023-12-25T20:52:38.140 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-25T20:52:38.445 INFO:teuthology.orchestra.run.smithi153.stdout:47244640261 2023-12-25T20:52:38.459 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640261 for osd.2 2023-12-25T20:52:38.459 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-25T20:52:38.749 INFO:teuthology.orchestra.run.smithi153.stdout:47244640260 2023-12-25T20:52:38.763 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640260 for osd.3 2023-12-25T20:52:38.763 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-25T20:52:39.053 INFO:teuthology.orchestra.run.smithi153.stdout:47244640260 2023-12-25T20:52:39.067 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640260 for osd.4 2023-12-25T20:52:39.067 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-25T20:52:39.358 INFO:teuthology.orchestra.run.smithi153.stdout:47244640260 2023-12-25T20:52:39.372 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640260 for osd.5 2023-12-25T20:52:39.372 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-25T20:52:39.373 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-25T20:52:39.661 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:52:39.663 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-12-25T20:52:39.674 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-12-25T20:52:37.734734+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1783416,"kb_used_data":840,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564447624,"statfs":{"total":579820584960,"available":577994366976,"internally_reserved":0,"allocated":860160,"data_stored":263454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.298411+0000","last_change":"2023-12-25T20:52:34.297698+0000","last_active":"2023-12-25T20:52:34.298411+0000","last_peered":"2023-12-25T20:52:34.298411+0000","last_clean":"2023-12-25T20:52:34.298411+0000","last_became_active":"2023-12-25T20:52:34.297388+0000","last_became_peered":"2023-12-25T20:52:34.297388+0000","last_unstale":"2023-12-25T20:52:34.298411+0000","last_undegraded":"2023-12-25T20:52:34.298411+0000","last_fullsized":"2023-12-25T20:52:34.298411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.297262+0000","last_change":"2023-12-25T20:52:34.294057+0000","last_active":"2023-12-25T20:52:34.297262+0000","last_peered":"2023-12-25T20:52:34.297262+0000","last_clean":"2023-12-25T20:52:34.297262+0000","last_became_active":"2023-12-25T20:52:34.293804+0000","last_became_peered":"2023-12-25T20:52:34.293804+0000","last_unstale":"2023-12-25T20:52:34.297262+0000","last_undegraded":"2023-12-25T20:52:34.297262+0000","last_fullsized":"2023-12-25T20:52:34.297262+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.301385+0000","last_change":"2023-12-25T20:52:34.300891+0000","last_active":"2023-12-25T20:52:34.301385+0000","last_peered":"2023-12-25T20:52:34.301385+0000","last_clean":"2023-12-25T20:52:34.301385+0000","last_became_active":"2023-12-25T20:52:34.300607+0000","last_became_peered":"2023-12-25T20:52:34.300607+0000","last_unstale":"2023-12-25T20:52:34.301385+0000","last_undegraded":"2023-12-25T20:52:34.301385+0000","last_fullsized":"2023-12-25T20:52:34.301385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.296213+0000","last_change":"2023-12-25T20:52:34.294456+0000","last_active":"2023-12-25T20:52:34.296213+0000","last_peered":"2023-12-25T20:52:34.296213+0000","last_clean":"2023-12-25T20:52:34.296213+0000","last_became_active":"2023-12-25T20:52:34.294123+0000","last_became_peered":"2023-12-25T20:52:34.294123+0000","last_unstale":"2023-12-25T20:52:34.296213+0000","last_undegraded":"2023-12-25T20:52:34.296213+0000","last_fullsized":"2023-12-25T20:52:34.296213+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.294693+0000","last_change":"2023-12-25T20:52:33.421321+0000","last_active":"2023-12-25T20:52:34.294693+0000","last_peered":"2023-12-25T20:52:34.294693+0000","last_clean":"2023-12-25T20:52:34.294693+0000","last_became_active":"2023-12-25T20:52:33.421006+0000","last_became_peered":"2023-12-25T20:52:33.421006+0000","last_unstale":"2023-12-25T20:52:34.294693+0000","last_undegraded":"2023-12-25T20:52:34.294693+0000","last_fullsized":"2023-12-25T20:52:34.294693+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:32.273733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:32.273733+0000","last_clean_scrub_stamp":"2023-12-25T20:52:32.273733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.299140+0000","last_change":"2023-12-25T20:52:34.297698+0000","last_active":"2023-12-25T20:52:34.299140+0000","last_peered":"2023-12-25T20:52:34.299140+0000","last_clean":"2023-12-25T20:52:34.299140+0000","last_became_active":"2023-12-25T20:52:34.297376+0000","last_became_peered":"2023-12-25T20:52:34.297376+0000","last_unstale":"2023-12-25T20:52:34.299140+0000","last_undegraded":"2023-12-25T20:52:34.299140+0000","last_fullsized":"2023-12-25T20:52:34.299140+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.305885+0000","last_change":"2023-12-25T20:52:34.300668+0000","last_active":"2023-12-25T20:52:34.305885+0000","last_peered":"2023-12-25T20:52:34.305885+0000","last_clean":"2023-12-25T20:52:34.305885+0000","last_became_active":"2023-12-25T20:52:34.300344+0000","last_became_peered":"2023-12-25T20:52:34.300344+0000","last_unstale":"2023-12-25T20:52:34.305885+0000","last_undegraded":"2023-12-25T20:52:34.305885+0000","last_fullsized":"2023-12-25T20:52:34.305885+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.297012+0000","last_change":"2023-12-25T20:52:34.295842+0000","last_active":"2023-12-25T20:52:34.297012+0000","last_peered":"2023-12-25T20:52:34.297012+0000","last_clean":"2023-12-25T20:52:34.297012+0000","last_became_active":"2023-12-25T20:52:34.295559+0000","last_became_peered":"2023-12-25T20:52:34.295559+0000","last_unstale":"2023-12-25T20:52:34.297012+0000","last_undegraded":"2023-12-25T20:52:34.297012+0000","last_fullsized":"2023-12-25T20:52:34.297012+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.305868+0000","last_change":"2023-12-25T20:52:34.300748+0000","last_active":"2023-12-25T20:52:34.305868+0000","last_peered":"2023-12-25T20:52:34.305868+0000","last_clean":"2023-12-25T20:52:34.305868+0000","last_became_active":"2023-12-25T20:52:34.300369+0000","last_became_peered":"2023-12-25T20:52:34.300369+0000","last_unstale":"2023-12-25T20:52:34.305868+0000","last_undegraded":"2023-12-25T20:52:34.305868+0000","last_fullsized":"2023-12-25T20:52:34.305868+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.056}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0249999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9500000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]}]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-25T20:52:39.675 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-25T20:52:39.962 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:52:39.964 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-12-25T20:52:39.975 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2023-12-25T20:52:39.735072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1783416,"kb_used_data":840,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564447624,"statfs":{"total":579820584960,"available":577994366976,"internally_reserved":0,"allocated":860160,"data_stored":263454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.298411+0000","last_change":"2023-12-25T20:52:34.297698+0000","last_active":"2023-12-25T20:52:34.298411+0000","last_peered":"2023-12-25T20:52:34.298411+0000","last_clean":"2023-12-25T20:52:34.298411+0000","last_became_active":"2023-12-25T20:52:34.297388+0000","last_became_peered":"2023-12-25T20:52:34.297388+0000","last_unstale":"2023-12-25T20:52:34.298411+0000","last_undegraded":"2023-12-25T20:52:34.298411+0000","last_fullsized":"2023-12-25T20:52:34.298411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.297262+0000","last_change":"2023-12-25T20:52:34.294057+0000","last_active":"2023-12-25T20:52:34.297262+0000","last_peered":"2023-12-25T20:52:34.297262+0000","last_clean":"2023-12-25T20:52:34.297262+0000","last_became_active":"2023-12-25T20:52:34.293804+0000","last_became_peered":"2023-12-25T20:52:34.293804+0000","last_unstale":"2023-12-25T20:52:34.297262+0000","last_undegraded":"2023-12-25T20:52:34.297262+0000","last_fullsized":"2023-12-25T20:52:34.297262+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.301385+0000","last_change":"2023-12-25T20:52:34.300891+0000","last_active":"2023-12-25T20:52:34.301385+0000","last_peered":"2023-12-25T20:52:34.301385+0000","last_clean":"2023-12-25T20:52:34.301385+0000","last_became_active":"2023-12-25T20:52:34.300607+0000","last_became_peered":"2023-12-25T20:52:34.300607+0000","last_unstale":"2023-12-25T20:52:34.301385+0000","last_undegraded":"2023-12-25T20:52:34.301385+0000","last_fullsized":"2023-12-25T20:52:34.301385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.296213+0000","last_change":"2023-12-25T20:52:34.294456+0000","last_active":"2023-12-25T20:52:34.296213+0000","last_peered":"2023-12-25T20:52:34.296213+0000","last_clean":"2023-12-25T20:52:34.296213+0000","last_became_active":"2023-12-25T20:52:34.294123+0000","last_became_peered":"2023-12-25T20:52:34.294123+0000","last_unstale":"2023-12-25T20:52:34.296213+0000","last_undegraded":"2023-12-25T20:52:34.296213+0000","last_fullsized":"2023-12-25T20:52:34.296213+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.294693+0000","last_change":"2023-12-25T20:52:33.421321+0000","last_active":"2023-12-25T20:52:34.294693+0000","last_peered":"2023-12-25T20:52:34.294693+0000","last_clean":"2023-12-25T20:52:34.294693+0000","last_became_active":"2023-12-25T20:52:33.421006+0000","last_became_peered":"2023-12-25T20:52:33.421006+0000","last_unstale":"2023-12-25T20:52:34.294693+0000","last_undegraded":"2023-12-25T20:52:34.294693+0000","last_fullsized":"2023-12-25T20:52:34.294693+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:32.273733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:32.273733+0000","last_clean_scrub_stamp":"2023-12-25T20:52:32.273733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.299140+0000","last_change":"2023-12-25T20:52:34.297698+0000","last_active":"2023-12-25T20:52:34.299140+0000","last_peered":"2023-12-25T20:52:34.299140+0000","last_clean":"2023-12-25T20:52:34.299140+0000","last_became_active":"2023-12-25T20:52:34.297376+0000","last_became_peered":"2023-12-25T20:52:34.297376+0000","last_unstale":"2023-12-25T20:52:34.299140+0000","last_undegraded":"2023-12-25T20:52:34.299140+0000","last_fullsized":"2023-12-25T20:52:34.299140+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.305885+0000","last_change":"2023-12-25T20:52:34.300668+0000","last_active":"2023-12-25T20:52:34.305885+0000","last_peered":"2023-12-25T20:52:34.305885+0000","last_clean":"2023-12-25T20:52:34.305885+0000","last_became_active":"2023-12-25T20:52:34.300344+0000","last_became_peered":"2023-12-25T20:52:34.300344+0000","last_unstale":"2023-12-25T20:52:34.305885+0000","last_undegraded":"2023-12-25T20:52:34.305885+0000","last_fullsized":"2023-12-25T20:52:34.305885+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.297012+0000","last_change":"2023-12-25T20:52:34.295842+0000","last_active":"2023-12-25T20:52:34.297012+0000","last_peered":"2023-12-25T20:52:34.297012+0000","last_clean":"2023-12-25T20:52:34.297012+0000","last_became_active":"2023-12-25T20:52:34.295559+0000","last_became_peered":"2023-12-25T20:52:34.295559+0000","last_unstale":"2023-12-25T20:52:34.297012+0000","last_undegraded":"2023-12-25T20:52:34.297012+0000","last_fullsized":"2023-12-25T20:52:34.297012+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.305868+0000","last_change":"2023-12-25T20:52:34.300748+0000","last_active":"2023-12-25T20:52:34.305868+0000","last_peered":"2023-12-25T20:52:34.305868+0000","last_clean":"2023-12-25T20:52:34.305868+0000","last_became_active":"2023-12-25T20:52:34.300369+0000","last_became_peered":"2023-12-25T20:52:34.300369+0000","last_unstale":"2023-12-25T20:52:34.305868+0000","last_undegraded":"2023-12-25T20:52:34.305868+0000","last_fullsized":"2023-12-25T20:52:34.305868+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.056}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0249999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9500000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]}]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-25T20:52:39.976 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-25T20:52:39.976 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-25T20:52:39.976 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-25T20:52:39.976 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-25T20:52:40.301 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:52:40.315 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-25T20:52:40.315 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-25T20:52:40.316 INFO:teuthology.run_tasks:Running task mon_thrash... 2023-12-25T20:52:40.328 INFO:tasks.mon_thrash:Beginning mon_thrash... 2023-12-25T20:52:40.328 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-25T20:52:40.619 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:52:40.619 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":15,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","created":"2023-12-25T20:52:22.419978+0000","modified":"2023-12-25T20:52:35.289112+0000","last_up_change":"2023-12-25T20:52:31.268017+0000","last_in_change":"2023-12-25T20:52:24.636112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-25T20:52:31.304424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-25T20:52:32.915209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"595d263d-6a56-4ccc-bb15-cd8d72e49f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":114394},{"type":"v1","addr":"172.21.15.153:6801","nonce":114394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":114394},{"type":"v1","addr":"172.21.15.153:6803","nonce":114394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":114394},{"type":"v1","addr":"172.21.15.153:6807","nonce":114394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":114394},{"type":"v1","addr":"172.21.15.153:6805","nonce":114394}]},"public_addr":"172.21.15.153:6801/114394","cluster_addr":"172.21.15.153:6803/114394","heartbeat_back_addr":"172.21.15.153:6807/114394","heartbeat_front_addr":"172.21.15.153:6805/114394","state":["exists","up"]},{"osd":1,"uuid":"3b8105a7-7f13-49a6-9388-23a4d9d131e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":114395},{"type":"v1","addr":"172.21.15.153:6809","nonce":114395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":114395},{"type":"v1","addr":"172.21.15.153:6811","nonce":114395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":114395},{"type":"v1","addr":"172.21.15.153:6815","nonce":114395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":114395},{"type":"v1","addr":"172.21.15.153:6813","nonce":114395}]},"public_addr":"172.21.15.153:6809/114395","cluster_addr":"172.21.15.153:6811/114395","heartbeat_back_addr":"172.21.15.153:6815/114395","heartbeat_front_addr":"172.21.15.153:6813/114395","state":["exists","up"]},{"osd":2,"uuid":"6b62f25d-a6fa-4089-9fd1-1b08d2cbad57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":114396},{"type":"v1","addr":"172.21.15.153:6817","nonce":114396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":114396},{"type":"v1","addr":"172.21.15.153:6819","nonce":114396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":114396},{"type":"v1","addr":"172.21.15.153:6823","nonce":114396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":114396},{"type":"v1","addr":"172.21.15.153:6821","nonce":114396}]},"public_addr":"172.21.15.153:6817/114396","cluster_addr":"172.21.15.153:6819/114396","heartbeat_back_addr":"172.21.15.153:6823/114396","heartbeat_front_addr":"172.21.15.153:6821/114396","state":["exists","up"]},{"osd":3,"uuid":"751e8edf-7ff9-4b73-9e8d-38416db248a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":114428},{"type":"v1","addr":"172.21.15.191:6801","nonce":114428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":114428},{"type":"v1","addr":"172.21.15.191:6803","nonce":114428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":114428},{"type":"v1","addr":"172.21.15.191:6807","nonce":114428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":114428},{"type":"v1","addr":"172.21.15.191:6805","nonce":114428}]},"public_addr":"172.21.15.191:6801/114428","cluster_addr":"172.21.15.191:6803/114428","heartbeat_back_addr":"172.21.15.191:6807/114428","heartbeat_front_addr":"172.21.15.191:6805/114428","state":["exists","up"]},{"osd":4,"uuid":"7635bbb7-ef87-483c-aadd-1984f330944c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":114429},{"type":"v1","addr":"172.21.15.191:6809","nonce":114429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":114429},{"type":"v1","addr":"172.21.15.191:6811","nonce":114429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":114429},{"type":"v1","addr":"172.21.15.191:6815","nonce":114429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":114429},{"type":"v1","addr":"172.21.15.191:6813","nonce":114429}]},"public_addr":"172.21.15.191:6809/114429","cluster_addr":"172.21.15.191:6811/114429","heartbeat_back_addr":"172.21.15.191:6815/114429","heartbeat_front_addr":"172.21.15.191:6813/114429","state":["exists","up"]},{"osd":5,"uuid":"7519398e-167d-40be-847b-1bea65ef0ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":114430},{"type":"v1","addr":"172.21.15.191:6817","nonce":114430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":114430},{"type":"v1","addr":"172.21.15.191:6819","nonce":114430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":114430},{"type":"v1","addr":"172.21.15.191:6823","nonce":114430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":114430},{"type":"v1","addr":"172.21.15.191:6821","nonce":114430}]},"public_addr":"172.21.15.191:6817/114430","cluster_addr":"172.21.15.191:6819/114430","heartbeat_back_addr":"172.21.15.191:6823/114430","heartbeat_front_addr":"172.21.15.191:6821/114430","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.682269+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.746139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.820727+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.689025+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.837599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.746784+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-25T20:52:40.632 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-12-25T20:52:31.304424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-12-25T20:52:32.915209+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-12-25T20:52:40.633 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2023-12-25T20:52:40.925 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 1 2023-12-25T20:52:40.938 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-12-25T20:52:41.228 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 8 2023-12-25T20:52:41.241 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2023-12-25T20:52:41.241 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-25T20:52:41.530 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:52:41.531 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-12-25T20:52:41.542 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2023-12-25T20:52:39.735072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1783416,"kb_used_data":840,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564447624,"statfs":{"total":579820584960,"available":577994366976,"internally_reserved":0,"allocated":860160,"data_stored":263454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.298411+0000","last_change":"2023-12-25T20:52:34.297698+0000","last_active":"2023-12-25T20:52:34.298411+0000","last_peered":"2023-12-25T20:52:34.298411+0000","last_clean":"2023-12-25T20:52:34.298411+0000","last_became_active":"2023-12-25T20:52:34.297388+0000","last_became_peered":"2023-12-25T20:52:34.297388+0000","last_unstale":"2023-12-25T20:52:34.298411+0000","last_undegraded":"2023-12-25T20:52:34.298411+0000","last_fullsized":"2023-12-25T20:52:34.298411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.297262+0000","last_change":"2023-12-25T20:52:34.294057+0000","last_active":"2023-12-25T20:52:34.297262+0000","last_peered":"2023-12-25T20:52:34.297262+0000","last_clean":"2023-12-25T20:52:34.297262+0000","last_became_active":"2023-12-25T20:52:34.293804+0000","last_became_peered":"2023-12-25T20:52:34.293804+0000","last_unstale":"2023-12-25T20:52:34.297262+0000","last_undegraded":"2023-12-25T20:52:34.297262+0000","last_fullsized":"2023-12-25T20:52:34.297262+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.301385+0000","last_change":"2023-12-25T20:52:34.300891+0000","last_active":"2023-12-25T20:52:34.301385+0000","last_peered":"2023-12-25T20:52:34.301385+0000","last_clean":"2023-12-25T20:52:34.301385+0000","last_became_active":"2023-12-25T20:52:34.300607+0000","last_became_peered":"2023-12-25T20:52:34.300607+0000","last_unstale":"2023-12-25T20:52:34.301385+0000","last_undegraded":"2023-12-25T20:52:34.301385+0000","last_fullsized":"2023-12-25T20:52:34.301385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.296213+0000","last_change":"2023-12-25T20:52:34.294456+0000","last_active":"2023-12-25T20:52:34.296213+0000","last_peered":"2023-12-25T20:52:34.296213+0000","last_clean":"2023-12-25T20:52:34.296213+0000","last_became_active":"2023-12-25T20:52:34.294123+0000","last_became_peered":"2023-12-25T20:52:34.294123+0000","last_unstale":"2023-12-25T20:52:34.296213+0000","last_undegraded":"2023-12-25T20:52:34.296213+0000","last_fullsized":"2023-12-25T20:52:34.296213+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.294693+0000","last_change":"2023-12-25T20:52:33.421321+0000","last_active":"2023-12-25T20:52:34.294693+0000","last_peered":"2023-12-25T20:52:34.294693+0000","last_clean":"2023-12-25T20:52:34.294693+0000","last_became_active":"2023-12-25T20:52:33.421006+0000","last_became_peered":"2023-12-25T20:52:33.421006+0000","last_unstale":"2023-12-25T20:52:34.294693+0000","last_undegraded":"2023-12-25T20:52:34.294693+0000","last_fullsized":"2023-12-25T20:52:34.294693+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:32.273733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:32.273733+0000","last_clean_scrub_stamp":"2023-12-25T20:52:32.273733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.299140+0000","last_change":"2023-12-25T20:52:34.297698+0000","last_active":"2023-12-25T20:52:34.299140+0000","last_peered":"2023-12-25T20:52:34.299140+0000","last_clean":"2023-12-25T20:52:34.299140+0000","last_became_active":"2023-12-25T20:52:34.297376+0000","last_became_peered":"2023-12-25T20:52:34.297376+0000","last_unstale":"2023-12-25T20:52:34.299140+0000","last_undegraded":"2023-12-25T20:52:34.299140+0000","last_fullsized":"2023-12-25T20:52:34.299140+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.305885+0000","last_change":"2023-12-25T20:52:34.300668+0000","last_active":"2023-12-25T20:52:34.305885+0000","last_peered":"2023-12-25T20:52:34.305885+0000","last_clean":"2023-12-25T20:52:34.305885+0000","last_became_active":"2023-12-25T20:52:34.300344+0000","last_became_peered":"2023-12-25T20:52:34.300344+0000","last_unstale":"2023-12-25T20:52:34.305885+0000","last_undegraded":"2023-12-25T20:52:34.305885+0000","last_fullsized":"2023-12-25T20:52:34.305885+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.297012+0000","last_change":"2023-12-25T20:52:34.295842+0000","last_active":"2023-12-25T20:52:34.297012+0000","last_peered":"2023-12-25T20:52:34.297012+0000","last_clean":"2023-12-25T20:52:34.297012+0000","last_became_active":"2023-12-25T20:52:34.295559+0000","last_became_peered":"2023-12-25T20:52:34.295559+0000","last_unstale":"2023-12-25T20:52:34.297012+0000","last_undegraded":"2023-12-25T20:52:34.297012+0000","last_fullsized":"2023-12-25T20:52:34.297012+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.305868+0000","last_change":"2023-12-25T20:52:34.300748+0000","last_active":"2023-12-25T20:52:34.305868+0000","last_peered":"2023-12-25T20:52:34.305868+0000","last_clean":"2023-12-25T20:52:34.305868+0000","last_became_active":"2023-12-25T20:52:34.300369+0000","last_became_peered":"2023-12-25T20:52:34.300369+0000","last_unstale":"2023-12-25T20:52:34.305868+0000","last_undegraded":"2023-12-25T20:52:34.305868+0000","last_fullsized":"2023-12-25T20:52:34.305868+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.056}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0249999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9500000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]}]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-25T20:52:41.543 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-25T20:52:41.833 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:52:41.834 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-12-25T20:52:41.845 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2023-12-25T20:52:41.735488+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1783416,"kb_used_data":840,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564447624,"statfs":{"total":579820584960,"available":577994366976,"internally_reserved":0,"allocated":860160,"data_stored":263454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.298411+0000","last_change":"2023-12-25T20:52:34.297698+0000","last_active":"2023-12-25T20:52:34.298411+0000","last_peered":"2023-12-25T20:52:34.298411+0000","last_clean":"2023-12-25T20:52:34.298411+0000","last_became_active":"2023-12-25T20:52:34.297388+0000","last_became_peered":"2023-12-25T20:52:34.297388+0000","last_unstale":"2023-12-25T20:52:34.298411+0000","last_undegraded":"2023-12-25T20:52:34.298411+0000","last_fullsized":"2023-12-25T20:52:34.298411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.297262+0000","last_change":"2023-12-25T20:52:34.294057+0000","last_active":"2023-12-25T20:52:34.297262+0000","last_peered":"2023-12-25T20:52:34.297262+0000","last_clean":"2023-12-25T20:52:34.297262+0000","last_became_active":"2023-12-25T20:52:34.293804+0000","last_became_peered":"2023-12-25T20:52:34.293804+0000","last_unstale":"2023-12-25T20:52:34.297262+0000","last_undegraded":"2023-12-25T20:52:34.297262+0000","last_fullsized":"2023-12-25T20:52:34.297262+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.301385+0000","last_change":"2023-12-25T20:52:34.300891+0000","last_active":"2023-12-25T20:52:34.301385+0000","last_peered":"2023-12-25T20:52:34.301385+0000","last_clean":"2023-12-25T20:52:34.301385+0000","last_became_active":"2023-12-25T20:52:34.300607+0000","last_became_peered":"2023-12-25T20:52:34.300607+0000","last_unstale":"2023-12-25T20:52:34.301385+0000","last_undegraded":"2023-12-25T20:52:34.301385+0000","last_fullsized":"2023-12-25T20:52:34.301385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.296213+0000","last_change":"2023-12-25T20:52:34.294456+0000","last_active":"2023-12-25T20:52:34.296213+0000","last_peered":"2023-12-25T20:52:34.296213+0000","last_clean":"2023-12-25T20:52:34.296213+0000","last_became_active":"2023-12-25T20:52:34.294123+0000","last_became_peered":"2023-12-25T20:52:34.294123+0000","last_unstale":"2023-12-25T20:52:34.296213+0000","last_undegraded":"2023-12-25T20:52:34.296213+0000","last_fullsized":"2023-12-25T20:52:34.296213+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.294693+0000","last_change":"2023-12-25T20:52:33.421321+0000","last_active":"2023-12-25T20:52:34.294693+0000","last_peered":"2023-12-25T20:52:34.294693+0000","last_clean":"2023-12-25T20:52:34.294693+0000","last_became_active":"2023-12-25T20:52:33.421006+0000","last_became_peered":"2023-12-25T20:52:33.421006+0000","last_unstale":"2023-12-25T20:52:34.294693+0000","last_undegraded":"2023-12-25T20:52:34.294693+0000","last_fullsized":"2023-12-25T20:52:34.294693+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:32.273733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:32.273733+0000","last_clean_scrub_stamp":"2023-12-25T20:52:32.273733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.299140+0000","last_change":"2023-12-25T20:52:34.297698+0000","last_active":"2023-12-25T20:52:34.299140+0000","last_peered":"2023-12-25T20:52:34.299140+0000","last_clean":"2023-12-25T20:52:34.299140+0000","last_became_active":"2023-12-25T20:52:34.297376+0000","last_became_peered":"2023-12-25T20:52:34.297376+0000","last_unstale":"2023-12-25T20:52:34.299140+0000","last_undegraded":"2023-12-25T20:52:34.299140+0000","last_fullsized":"2023-12-25T20:52:34.299140+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.305885+0000","last_change":"2023-12-25T20:52:34.300668+0000","last_active":"2023-12-25T20:52:34.305885+0000","last_peered":"2023-12-25T20:52:34.305885+0000","last_clean":"2023-12-25T20:52:34.305885+0000","last_became_active":"2023-12-25T20:52:34.300344+0000","last_became_peered":"2023-12-25T20:52:34.300344+0000","last_unstale":"2023-12-25T20:52:34.305885+0000","last_undegraded":"2023-12-25T20:52:34.305885+0000","last_fullsized":"2023-12-25T20:52:34.305885+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.297012+0000","last_change":"2023-12-25T20:52:34.295842+0000","last_active":"2023-12-25T20:52:34.297012+0000","last_peered":"2023-12-25T20:52:34.297012+0000","last_clean":"2023-12-25T20:52:34.297012+0000","last_became_active":"2023-12-25T20:52:34.295559+0000","last_became_peered":"2023-12-25T20:52:34.295559+0000","last_unstale":"2023-12-25T20:52:34.297012+0000","last_undegraded":"2023-12-25T20:52:34.297012+0000","last_fullsized":"2023-12-25T20:52:34.297012+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-25T20:52:34.305868+0000","last_change":"2023-12-25T20:52:34.300748+0000","last_active":"2023-12-25T20:52:34.305868+0000","last_peered":"2023-12-25T20:52:34.305868+0000","last_clean":"2023-12-25T20:52:34.305868+0000","last_became_active":"2023-12-25T20:52:34.300369+0000","last_became_peered":"2023-12-25T20:52:34.300369+0000","last_unstale":"2023-12-25T20:52:34.305868+0000","last_undegraded":"2023-12-25T20:52:34.305868+0000","last_fullsized":"2023-12-25T20:52:34.305868+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.056}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0249999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9500000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]}]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-25T20:52:41.846 INFO:tasks.mon_thrash.ceph_manager:clean! 2023-12-25T20:52:41.847 DEBUG:tasks.mon_thrash:Yielding 2023-12-25T20:52:41.847 INFO:teuthology.run_tasks:Running task workunit... 2023-12-25T20:52:41.866 INFO:tasks.workunit:Pulling workunits from ref b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 2023-12-25T20:52:41.866 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-25T20:52:41.867 DEBUG:teuthology.orchestra.run.smithi191:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-25T20:52:41.868 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2023-12-25T20:52:41.868 INFO:tasks.mon_thrash.mon_thrasher:seed: 1703537561, revive delay: 20.0, thrash delay: 1.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2023-12-25T20:52:41.868 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-25T20:52:41.868 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:52:41.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T20:52:41.891 INFO:teuthology.orchestra.run.smithi191.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-12-25T20:52:41.891 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-12-25T20:52:41.952 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-12-25T20:52:41.952 DEBUG:teuthology.orchestra.run.smithi191:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-12-25T20:52:42.012 INFO:tasks.workunit:timeout=3h 2023-12-25T20:52:42.013 INFO:tasks.workunit:cleanup=True 2023-12-25T20:52:42.013 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 2023-12-25T20:52:42.072 INFO:tasks.workunit.client.0.smithi191.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-25T20:52:42.221 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:52:42.235 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-25T20:52:42.235 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-12-25T20:52:42.235 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-25T20:52:42.235 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-25T20:52:42.375 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:52:42.375 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:52:42.375 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:52:42.375 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2023-12-25T20:52:42.376 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 8, 2023-12-25T20:52:42.376 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:52:42.376 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:52:42.376 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:52:42.376 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:52:42.376 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:42.376 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 19, 2023-12-25T20:52:42.376 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:52:42.376 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:52:42.376 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:52:42.376 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:52:42.376 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:52:42.376 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:52:42.376 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:52:42.376 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:52:42.377 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:52:42.377 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:52:42.377 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:52:42.377 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:42.377 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:52:42.377 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:52:42.377 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:52:42.377 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:52:42.377 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:52:42.377 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:52:42.377 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:52:42.377 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:52:42.378 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:52:42.378 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:52:42.378 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:42.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.378 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:52:42.378 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:52:42.378 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:52:42.378 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:52:42.378 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:52:42.378 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:52:42.378 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:52:42.379 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:52:42.379 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:52:42.379 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:52:42.379 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:52:42.379 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:52:42.379 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:52:42.379 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:52:42.379 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:52:42.379 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:52:42.379 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:52:42.379 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:52:42.379 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:52:42.379 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:52:42.379 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:52:42.379 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:52:42.380 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:52:42.380 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:52:42.380 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:52:42.380 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:42.380 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:52:42.380 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.380 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:52:42.380 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.380 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:52:42.380 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:52:42.380 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:52:42.380 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:52:42.380 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.380 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:52:42.380 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:52:42.381 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:42.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.381 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.381 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:52:42.381 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:52:42.381 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:42.381 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.381 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:42.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.381 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:52:42.381 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:52:42.381 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:52:42.381 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:52:42.382 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:52:42.382 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.382 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.382 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:52:42.382 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:52:42.382 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:52:42.382 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:52:42.382 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.382 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:52:42.382 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:52:42.382 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:42.382 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.382 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.382 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:52:42.382 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:52:42.383 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:42.383 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.383 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:42.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.383 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:52:42.383 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:52:42.383 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:52:42.383 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:52:42.383 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:52:42.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.383 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.383 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:52:42.383 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:52:42.383 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:52:42.383 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:52:42.384 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.384 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:52:42.384 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:52:42.384 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:42.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.384 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.384 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:52:42.384 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:52:42.384 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:42.384 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.384 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:42.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.384 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:52:42.384 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:52:42.385 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:52:42.385 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:52:42.385 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:52:42.385 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.385 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:42.385 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.385 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:52:42.385 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:52:42.385 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.385 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:52:42.385 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:52:42.386 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:52:42.386 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.386 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:42.386 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:52:42.386 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.386 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:52:42.386 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:52:42.386 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2023-12-25T20:52:42.386 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.386 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:42.386 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:52:42.387 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.387 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:52:42.387 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:52:42.387 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2023-12-25T20:52:42.387 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.387 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:42.387 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.387 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:52:42.387 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:52:42.388 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:52:42.388 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-25T20:52:42.388 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-25T20:52:42.524 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:52:42.525 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:52:42.525 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:52:42.525 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2023-12-25T20:52:42.525 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 8, 2023-12-25T20:52:42.525 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:52:42.525 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:52:42.525 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:52:42.525 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:52:42.526 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:42.526 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 20, 2023-12-25T20:52:42.526 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:52:42.526 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:52:42.526 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:52:42.526 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:52:42.526 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:52:42.526 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:52:42.526 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:52:42.526 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:52:42.526 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:52:42.526 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:52:42.526 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:52:42.526 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:42.527 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:52:42.527 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:52:42.527 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:52:42.527 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:52:42.527 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:52:42.527 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:52:42.527 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:52:42.527 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:52:42.527 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:52:42.527 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:52:42.527 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:42.527 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.527 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:52:42.527 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:52:42.527 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:52:42.528 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:52:42.528 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:52:42.528 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:52:42.528 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:52:42.528 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:52:42.528 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:52:42.528 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:52:42.528 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:52:42.528 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:52:42.528 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:52:42.528 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:52:42.528 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:52:42.528 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:52:42.528 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:52:42.528 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:52:42.529 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:52:42.529 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:52:42.529 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:52:42.529 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:52:42.529 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:52:42.529 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:52:42.529 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:52:42.529 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:42.529 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:52:42.529 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.529 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:52:42.529 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.529 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:52:42.530 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:52:42.530 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:52:42.530 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:52:42.530 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.530 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:52:42.530 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:52:42.530 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:42.530 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.530 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.530 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:52:42.530 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:52:42.530 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:42.530 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.530 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:42.531 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.531 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:52:42.531 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:52:42.531 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:52:42.531 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:52:42.531 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:52:42.531 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.531 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.531 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:52:42.531 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:52:42.531 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:52:42.531 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:52:42.532 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.532 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:52:42.532 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:52:42.532 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:42.532 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.532 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.532 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:52:42.532 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:52:42.532 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:42.533 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.533 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:42.533 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.533 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:52:42.533 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:52:42.533 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:52:42.533 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:52:42.533 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:52:42.533 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.533 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.533 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:52:42.533 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:52:42.533 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:52:42.533 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:52:42.533 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.534 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:52:42.534 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:52:42.534 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:42.534 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.534 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.534 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:52:42.534 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:52:42.534 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:42.535 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.535 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:42.535 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.535 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:52:42.535 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:52:42.535 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:52:42.535 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:52:42.535 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:52:42.535 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.535 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:42.535 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.535 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:52:42.535 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:52:42.535 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.535 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:52:42.535 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:52:42.536 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:52:42.536 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.536 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:42.536 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:52:42.536 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.536 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:52:42.536 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:52:42.536 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2023-12-25T20:52:42.536 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.536 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:42.536 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:52:42.536 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.536 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:52:42.536 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:52:42.536 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:52:42.537 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.537 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:42.537 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.537 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:52:42.537 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:52:42.538 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:52:42.538 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-25T20:52:42.538 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-25T20:52:42.676 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:52:42.677 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:52:42.677 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:52:42.677 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2023-12-25T20:52:42.677 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 8, 2023-12-25T20:52:42.677 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:52:42.677 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:52:42.677 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:52:42.677 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:52:42.677 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:42.677 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 20, 2023-12-25T20:52:42.677 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:52:42.677 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:52:42.678 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:52:42.678 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:52:42.678 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:52:42.678 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:52:42.678 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:52:42.678 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:52:42.678 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:52:42.678 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:52:42.678 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:52:42.678 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:42.678 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:52:42.678 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:52:42.678 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:52:42.678 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:52:42.679 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:52:42.679 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:52:42.679 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:52:42.679 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:52:42.679 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:52:42.679 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:52:42.679 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:42.679 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.679 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:52:42.679 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:52:42.679 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:52:42.679 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:52:42.679 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:52:42.679 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:52:42.679 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:52:42.680 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:52:42.680 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:52:42.680 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:52:42.680 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:52:42.680 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:52:42.680 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:52:42.680 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:52:42.680 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:52:42.680 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:52:42.680 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:52:42.680 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:52:42.680 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:52:42.680 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:52:42.680 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:52:42.680 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:52:42.681 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:52:42.681 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:52:42.681 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:52:42.681 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:42.681 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:52:42.681 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.681 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:52:42.681 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.681 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:52:42.681 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:52:42.681 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:52:42.681 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:52:42.681 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.681 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:52:42.681 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:52:42.682 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:42.682 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.682 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.682 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:52:42.682 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:52:42.682 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:42.682 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.682 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:42.682 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.682 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:52:42.682 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:52:42.682 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:52:42.682 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:52:42.682 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:52:42.683 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.683 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.683 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:52:42.683 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:52:42.683 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:52:42.683 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:52:42.683 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.683 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:52:42.683 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:52:42.683 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:42.683 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.683 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.683 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:52:42.683 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:52:42.683 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:42.683 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.684 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:42.684 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.684 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:52:42.684 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:52:42.684 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:52:42.684 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:52:42.684 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:52:42.684 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.684 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.684 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:52:42.684 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:52:42.684 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:52:42.684 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:52:42.684 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.684 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:52:42.684 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:52:42.685 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:42.685 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.685 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.685 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:52:42.685 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:52:42.685 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:42.685 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.685 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:42.685 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.685 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:52:42.685 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:52:42.685 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:52:42.685 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:52:42.685 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:52:42.686 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.686 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:42.686 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.686 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:52:42.686 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:52:42.686 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.686 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:52:42.686 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:52:42.686 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:52:42.686 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.686 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:42.686 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:52:42.686 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.686 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:52:42.686 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:52:42.686 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:52:42.687 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.687 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:42.687 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:52:42.687 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.687 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:52:42.687 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:52:42.687 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2023-12-25T20:52:42.687 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.687 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:42.687 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2023-12-25T20:52:42.687 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:42.687 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:52:42.687 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:52:42.687 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:52:42.687 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:42.687 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:42.687 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:42.688 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:52:42.688 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:52:42.688 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:52:42.689 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2023-12-25T20:52:42.689 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-12-25T20:52:42.689 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2023-12-25T20:52:42.689 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-12-25T20:52:42.689 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-25T20:52:42.689 INFO:teuthology.orchestra.run:waiting for 300 2023-12-25T20:52:42.700 INFO:tasks.ceph.mon.b:Stopped 2023-12-25T20:52:42.700 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-12-25T20:52:42.700 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:52:43.055 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:52:46.069 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:52:46.433 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:52:49.448 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:52:50.644 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":27,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:52:53.662 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:52:57.635 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":12,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:52:57.648 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-12-25T20:52:57.648 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-12-25T20:52:57.648 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-25T20:52:57.785 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:52:57.785 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:52:57.785 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:52:57.785 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2023-12-25T20:52:57.785 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 12, 2023-12-25T20:52:57.786 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:52:57.786 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:52:57.786 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:52:57.786 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:57.786 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2023-12-25T20:52:57.786 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:52:57.786 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:52:57.786 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:52:57.786 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:52:57.786 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:52:57.786 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:52:57.786 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:52:57.786 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:52:57.786 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:52:57.786 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:52:57.787 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:52:57.787 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:57.787 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:52:57.787 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:52:57.787 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:52:57.787 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:52:57.787 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:52:57.787 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:52:57.787 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:52:57.787 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:52:57.787 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:52:57.787 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:52:57.787 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:57.787 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.788 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:52:57.788 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:52:57.788 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:52:57.788 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:52:57.788 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:52:57.788 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:52:57.788 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:52:57.788 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:52:57.788 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:52:57.788 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:52:57.788 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:52:57.788 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:52:57.788 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:52:57.788 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:52:57.788 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:52:57.789 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:52:57.789 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:52:57.789 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:52:57.789 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:52:57.789 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:52:57.789 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:52:57.789 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:52:57.789 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:52:57.789 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:52:57.789 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:52:57.789 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:57.790 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:52:57.790 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.790 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:52:57.790 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.790 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:52:57.790 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:52:57.790 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:52:57.790 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:52:57.790 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.790 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:52:57.790 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:52:57.790 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:57.791 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.791 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.791 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:52:57.791 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:52:57.791 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:57.791 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:57.791 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:57.791 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.791 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:52:57.791 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:52:57.791 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:52:57.791 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:52:57.791 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:52:57.791 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.791 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.792 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:52:57.792 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:52:57.792 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:52:57.792 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:52:57.792 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.792 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:52:57.792 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:52:57.792 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:57.792 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.792 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.792 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:52:57.792 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:52:57.792 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:57.792 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:57.792 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:57.793 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.793 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:52:57.793 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:52:57.793 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:52:57.793 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:52:57.793 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:52:57.793 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.793 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.794 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:52:57.794 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:52:57.794 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:52:57.794 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:52:57.794 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.794 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:52:57.794 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:52:57.794 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:57.794 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.794 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.794 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:52:57.794 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:52:57.794 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:57.794 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:57.794 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:57.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.795 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:52:57.795 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:52:57.795 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:52:57.795 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:52:57.795 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:52:57.795 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:57.795 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:57.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.795 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:52:57.795 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:52:57.795 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.795 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:52:57.795 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:52:57.795 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:52:57.796 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:57.796 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:57.796 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:52:57.796 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.796 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:52:57.796 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:52:57.796 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2023-12-25T20:52:57.796 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:57.796 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:57.796 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:52:57.796 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.796 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:52:57.796 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:52:57.796 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2023-12-25T20:52:57.796 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:57.797 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:57.797 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2023-12-25T20:52:57.797 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.797 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:52:57.797 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:52:57.797 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:52:57.797 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:57.797 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:57.797 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.797 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:52:57.797 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:52:57.798 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-25T20:52:57.798 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-12-25T20:52:57.798 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-25T20:52:57.935 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:52:57.935 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:52:57.935 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:52:57.935 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2023-12-25T20:52:57.935 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 12, 2023-12-25T20:52:57.935 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:52:57.935 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:52:57.935 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:52:57.935 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:57.935 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2023-12-25T20:52:57.935 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:52:57.936 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:52:57.936 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:52:57.936 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:52:57.936 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:52:57.936 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:52:57.936 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:52:57.936 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:52:57.936 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:52:57.936 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:52:57.936 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:52:57.936 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:57.936 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:52:57.936 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:52:57.936 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:52:57.936 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:52:57.937 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:52:57.937 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:52:57.937 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:52:57.937 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:52:57.937 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:52:57.937 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:52:57.937 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:57.937 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.937 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:52:57.937 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:52:57.937 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:52:57.937 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:52:57.937 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:52:57.937 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:52:57.938 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:52:57.938 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:52:57.938 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:52:57.938 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:52:57.938 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:52:57.938 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:52:57.938 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:52:57.938 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:52:57.938 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:52:57.938 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:52:57.938 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:52:57.938 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:52:57.938 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:52:57.938 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:52:57.939 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:52:57.939 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:52:57.939 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:52:57.939 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:52:57.939 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:52:57.939 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:57.939 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:52:57.939 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.939 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:52:57.939 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.939 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:52:57.939 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:52:57.939 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:52:57.939 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:52:57.939 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.940 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:52:57.940 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:52:57.940 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:57.940 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.940 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.940 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:52:57.940 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:52:57.940 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:57.940 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:57.940 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:57.940 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.940 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:52:57.940 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:52:57.940 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:52:57.940 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:52:57.941 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:52:57.941 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.941 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.941 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:52:57.941 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:52:57.941 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:52:57.941 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:52:57.941 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.941 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:52:57.941 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:52:57.941 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:57.941 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.941 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.941 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:52:57.942 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:52:57.942 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:57.942 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:57.942 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:57.942 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.942 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:52:57.942 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:52:57.942 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:52:57.942 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:52:57.942 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:52:57.942 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.942 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.942 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:52:57.942 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:52:57.942 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:52:57.943 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:52:57.943 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.943 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:52:57.943 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:52:57.943 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:57.943 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.943 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.943 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:52:57.943 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:52:57.943 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:52:57.943 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:57.943 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:57.943 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.943 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:52:57.943 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:52:57.944 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:52:57.944 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:52:57.944 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:52:57.944 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:57.944 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:57.944 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.944 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:52:57.944 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:52:57.944 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.944 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:52:57.944 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:52:57.944 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:52:57.944 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:57.944 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:57.944 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:52:57.945 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.945 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:52:57.945 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:52:57.945 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2023-12-25T20:52:57.945 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:57.945 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:52:57.945 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:52:57.945 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:52:57.945 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:52:57.945 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:52:57.945 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:52:57.945 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:52:57.945 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:52:57.945 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:52:57.945 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:52:57.946 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:52:57.947 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-25T20:52:57.947 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-12-25T20:53:17.948 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-12-25T20:53:17.948 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2023-12-25T20:53:17.948 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-25T20:53:17.948 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-12-25T20:53:17.951 INFO:tasks.ceph.mon.b:Started 2023-12-25T20:53:17.951 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-25T20:53:17.951 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:53:23.092 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":16,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:53:26.111 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:53:26.479 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:53:26.493 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-25T20:53:26.493 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-25T20:53:26.493 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-25T20:53:26.832 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:53:26.832 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:53:26.832 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:53:26.832 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2023-12-25T20:53:26.833 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 18, 2023-12-25T20:53:26.833 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:53:26.833 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:53:26.833 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:53:26.833 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:53:26.833 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:26.833 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2023-12-25T20:53:26.833 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:53:26.833 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:53:26.833 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:53:26.833 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:26.833 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:26.833 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:26.833 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:26.834 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:26.834 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:26.834 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:26.834 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:26.834 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:26.834 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:53:26.834 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:53:26.834 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:26.834 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:26.834 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:26.834 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:26.834 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:26.834 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:26.834 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:26.835 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:26.835 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:26.835 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.835 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:53:26.835 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:53:26.835 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:53:26.835 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:53:26.835 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:53:26.835 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:53:26.835 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:53:26.835 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:53:26.835 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:53:26.835 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:53:26.835 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:53:26.835 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:53:26.835 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:53:26.836 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:53:26.836 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:53:26.836 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:53:26.836 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:53:26.836 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:26.836 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:26.836 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:26.836 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:26.836 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:26.836 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:26.836 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:26.836 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:26.836 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:26.836 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:53:26.837 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.837 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:53:26.837 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.837 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:53:26.837 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:53:26.837 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:26.837 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:26.837 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.837 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:26.837 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:53:26.837 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:26.837 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.837 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.837 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:26.837 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:53:26.838 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:26.838 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:26.838 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:26.838 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.838 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:53:26.838 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:53:26.838 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:26.838 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:26.838 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:26.838 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.838 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.838 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:53:26.838 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:53:26.838 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:26.838 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:26.839 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.839 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:26.839 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:53:26.839 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:26.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.839 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.839 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:26.839 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:53:26.839 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:26.839 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:26.839 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:26.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.839 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:53:26.839 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:53:26.839 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:26.840 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:26.840 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:26.840 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.840 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.840 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:53:26.840 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:53:26.840 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:26.840 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:26.840 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.840 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:26.840 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:53:26.840 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:26.840 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.840 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.840 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:26.841 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:53:26.841 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:26.841 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:26.841 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:26.841 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.841 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:53:26.841 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:53:26.841 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:26.841 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:26.841 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:26.841 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:26.841 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:26.841 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.841 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:53:26.841 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:53:26.842 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.842 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:26.842 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:26.842 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:53:26.842 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:26.842 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:26.842 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:53:26.842 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.842 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:26.842 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:26.842 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2023-12-25T20:53:26.842 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:26.842 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:26.842 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:53:26.842 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.843 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:26.843 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:26.843 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2023-12-25T20:53:26.843 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:26.843 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:26.843 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2023-12-25T20:53:26.843 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.843 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:26.843 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:26.843 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:53:26.843 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:26.843 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:26.843 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.843 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:53:26.843 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:53:26.844 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:53:26.845 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-25T20:53:26.845 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-25T20:53:26.985 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:53:26.985 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:53:26.985 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:53:26.985 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2023-12-25T20:53:26.986 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 18, 2023-12-25T20:53:26.986 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:53:26.986 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:53:26.986 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:53:26.986 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:53:26.986 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:26.986 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2023-12-25T20:53:26.986 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:53:26.986 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:53:26.986 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:53:26.986 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:26.986 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:26.986 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:26.986 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:26.986 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:26.987 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:26.987 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:26.987 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:26.987 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:26.987 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:53:26.987 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:53:26.987 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:26.987 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:26.987 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:26.987 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:26.987 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:26.987 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:26.987 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:26.987 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:26.987 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:26.988 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.988 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:53:26.988 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:53:26.988 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:53:26.988 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:53:26.988 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:53:26.988 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:53:26.988 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:53:26.988 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:53:26.988 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:53:26.988 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:53:26.988 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:53:26.988 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:53:26.988 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:53:26.988 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:53:26.988 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:53:26.989 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:53:26.989 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:53:26.989 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:26.989 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:26.989 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:26.989 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:26.989 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:26.989 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:26.989 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:26.989 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:26.989 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:26.989 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:53:26.989 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.989 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:53:26.989 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.990 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:53:26.990 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:53:26.990 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:26.990 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:26.990 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.990 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:26.990 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:53:26.990 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:26.990 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.990 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.990 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:26.990 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:53:26.990 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:26.990 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:26.990 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:26.990 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.991 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:53:26.991 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:53:26.991 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:26.991 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:26.991 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:26.991 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.991 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.991 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:53:26.991 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:53:26.991 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:26.991 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:26.991 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.991 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:26.991 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:53:26.991 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:26.992 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.992 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.992 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:26.992 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:53:26.992 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:26.992 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:26.992 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:26.992 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.992 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:53:26.992 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:53:26.992 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:26.992 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:26.992 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:26.992 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.992 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.992 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:53:26.993 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:53:26.993 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:26.993 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:26.993 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.993 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:26.993 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:53:26.993 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:26.993 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.993 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.993 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:26.993 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:53:26.993 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:26.993 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:26.993 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:26.993 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.994 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:53:26.994 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:53:26.994 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:26.994 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:26.994 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:26.994 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:26.994 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:26.994 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.994 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:53:26.994 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:53:26.994 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.994 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:26.994 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:26.994 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:53:26.994 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:26.994 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:26.995 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:53:26.995 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.995 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:26.995 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:26.995 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2023-12-25T20:53:26.995 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:26.995 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:26.995 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:53:26.995 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:26.995 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:26.995 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:26.995 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2023-12-25T20:53:26.995 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:26.995 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:26.995 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:26.995 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:53:26.996 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:53:26.997 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:53:26.997 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-25T20:53:26.997 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-25T20:53:27.136 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:53:27.136 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:53:27.136 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:53:27.136 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2023-12-25T20:53:27.136 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 18, 2023-12-25T20:53:27.136 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:53:27.136 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:53:27.136 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:53:27.136 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:53:27.136 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:27.136 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2023-12-25T20:53:27.137 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:53:27.137 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:53:27.137 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:53:27.137 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:27.137 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:27.137 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:27.137 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:27.137 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:27.137 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:27.137 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:27.137 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:27.137 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:27.137 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:53:27.137 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:53:27.137 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:27.138 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:27.138 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:27.138 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:27.138 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:27.138 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:27.138 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:27.138 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:27.138 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:27.138 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:27.138 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:53:27.138 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:53:27.138 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:53:27.138 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:53:27.138 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:53:27.138 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:53:27.139 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:53:27.139 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:53:27.139 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:53:27.139 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:53:27.139 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:53:27.139 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:53:27.139 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:53:27.139 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:53:27.139 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:53:27.139 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:53:27.139 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:53:27.139 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:27.139 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:27.139 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:27.139 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:27.139 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:27.140 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:27.140 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:27.140 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:27.140 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:27.140 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:53:27.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:27.140 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:53:27.140 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:27.140 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:53:27.140 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:53:27.140 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:27.140 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:27.140 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:27.140 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:27.140 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:53:27.141 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:27.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:27.141 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:27.141 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:27.141 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:53:27.141 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:27.141 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:27.141 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:27.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:27.141 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:53:27.141 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:53:27.141 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:27.141 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:27.141 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:27.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:27.142 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:27.142 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:53:27.142 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:53:27.142 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:27.142 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:27.142 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:27.142 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:27.142 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:53:27.142 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:27.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:27.142 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:27.142 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:27.142 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:53:27.142 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:27.142 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:27.143 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:27.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:27.143 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:53:27.143 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:53:27.143 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:27.143 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:27.143 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:27.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:27.144 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:27.144 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:53:27.144 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:53:27.144 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:27.144 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:27.144 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:27.144 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:27.144 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:53:27.144 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:27.144 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:27.144 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:27.144 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:27.144 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:53:27.144 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:27.144 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:27.145 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:27.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:27.145 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:53:27.145 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:53:27.145 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:27.145 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:27.145 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:27.145 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:27.145 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:27.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:27.145 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:53:27.145 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:53:27.145 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:27.145 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:27.145 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:27.146 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:53:27.146 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:27.146 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:27.146 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:53:27.146 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:27.146 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:27.146 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:27.146 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:53:27.146 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:27.146 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:27.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:27.146 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:53:27.146 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:53:27.147 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:53:27.147 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-12-25T20:53:27.148 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-12-25T20:53:27.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:53:27.507+0000 7f7a902f1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:53:27.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:53:27.507+0000 7f7a902f1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=6,logm=66,mdsmap=3,mgr=1} crc {auth=361746278,config=3140514009,health=3806841325,kv=661943861,logm=3878195515,mdsmap=1531151339,mgr=2835230805}) 2023-12-25T20:53:27.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:53:27.507+0000 7f7a902f1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=11,kv=5,logm=66,mdsmap=3,mgr=3} crc {auth=361746278,config=3140514009,health=3035693388,kv=661943861,logm=455555201,mdsmap=1531151339,mgr=801034601}) 2023-12-25T20:53:27.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:53:27.507+0000 7f7a902f1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:53:27.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:53:27.507+0000 7f7a902f1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=6,logm=66,mdsmap=3,mgr=1} crc {auth=361746278,config=3140514009,health=3806841325,kv=661943861,logm=3878195515,mdsmap=1531151339,mgr=2835230805}) 2023-12-25T20:53:27.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:53:27.507+0000 7f7a902f1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=9,config=1,health=10,kv=6,logm=67,mdsmap=3,mgr=4} crc {auth=932004695,config=1228700967,health=1057758512,kv=661943861,logm=1777284046,mdsmap=2147943554,mgr=1426436156}) 2023-12-25T20:53:27.508 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:53:27.508+0000 7f7a902f1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:53:27.508 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:53:27.508+0000 7f7a902f1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=6,mgr_metadata=1,mgrstat=10,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=28} crc {mgr=3698185029,mgr_metadata=1482708640,mgrstat=2757287655,mon_config_key=2919671258,monmap=3296645973,osd_metadata=1683759159,osd_pg_creating=534015559,osdmap=3480423978}) 2023-12-25T20:53:27.508 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:53:27.508+0000 7f7a902f1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=3,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=32} crc {mgr=3698185029,mgr_command_descs=1448462394,mgr_metadata=1482708640,mgrstat=2757287655,mon_config_key=2934451455,monmap=3296645973,osd_metadata=2299837211,osd_pg_creating=534015559,osdmap=818453809}) 2023-12-25T20:53:27.508 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:53:27.508+0000 7f7a902f1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:53:27.509 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:53:27.508+0000 7f7a902f1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=6,mgr_metadata=1,mgrstat=10,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=28} crc {mgr=3698185029,mgr_metadata=1482708640,mgrstat=2757287655,mon_config_key=2919671258,monmap=3296645973,osd_metadata=1683759159,osd_pg_creating=534015559,osdmap=3480423978}) 2023-12-25T20:53:27.509 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:53:27.508+0000 7f7a902f1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=3,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=31} crc {mgr=3698185029,mgr_command_descs=1448462394,mgr_metadata=1482708640,mgrstat=2757287655,mon_config_key=2934451455,monmap=3296645973,osd_metadata=3245185047,osd_pg_creating=534015560,osdmap=2122473917}) 2023-12-25T20:53:27.520 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-12-25T20:53:28.521 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-25T20:53:28.521 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:53:28.887 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:53:28.901 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-25T20:53:28.901 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-12-25T20:53:28.901 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-25T20:53:28.901 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-25T20:53:29.038 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:53:29.038 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:53:29.038 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:53:29.038 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2023-12-25T20:53:29.038 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 18, 2023-12-25T20:53:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:53:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:53:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:53:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:53:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2023-12-25T20:53:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:53:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:53:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:53:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:53:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:53:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:53:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:53:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:53:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:53:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:53:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:53:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:53:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:53:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:53:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:53:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:53:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:53:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:53:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:53:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:53:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:53:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:53:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:29.043 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:29.043 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:29.043 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:53:29.043 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.043 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:53:29.043 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.043 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:53:29.043 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:53:29.043 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:29.043 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:29.043 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.043 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:29.043 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:53:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:53:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:53:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:53:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:29.045 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:29.045 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.045 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.045 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:53:29.045 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:53:29.045 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:29.045 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:29.045 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.046 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:29.046 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:53:29.046 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:29.046 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.046 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.046 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:29.047 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:53:29.047 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:29.047 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:29.047 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:29.047 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.047 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:53:29.047 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:53:29.047 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:29.047 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:29.047 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:29.047 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.047 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.047 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:53:29.047 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:53:29.047 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:29.048 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:29.048 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.048 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:29.048 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:53:29.048 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:29.048 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.048 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.048 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:29.048 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:53:29.048 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:29.048 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:29.048 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:29.048 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.049 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:53:29.049 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:53:29.049 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:29.049 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:29.049 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:29.049 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:29.049 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:29.049 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.049 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:53:29.049 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:53:29.049 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.049 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:29.049 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:29.050 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:53:29.050 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:29.050 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:29.050 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:53:29.050 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.050 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:29.050 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:29.050 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2023-12-25T20:53:29.050 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:29.050 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:29.050 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:53:29.050 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.050 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:29.050 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:29.051 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2023-12-25T20:53:29.051 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:29.051 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:29.051 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2023-12-25T20:53:29.051 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.051 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:29.051 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:29.051 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:53:29.051 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:29.051 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:29.051 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.051 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:53:29.051 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:53:29.052 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:53:29.053 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-25T20:53:29.053 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-25T20:53:29.188 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:53:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:53:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:53:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2023-12-25T20:53:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 18, 2023-12-25T20:53:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:53:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:53:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:53:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:53:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2023-12-25T20:53:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:53:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:53:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:53:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:53:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:53:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:53:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:53:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:53:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:53:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:53:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:53:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:53:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:53:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:53:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:53:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:53:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:53:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:53:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:53:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:53:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:53:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:53:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:53:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:53:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:53:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:53:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:53:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:53:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:53:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:53:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:53:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:53:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:53:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:53:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:53:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:53:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:53:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:53:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:29.197 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:29.197 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.197 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:53:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:53:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:53:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:53:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:53:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:53:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:53:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:53:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2023-12-25T20:53:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:53:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:53:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:53:29.201 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:53:29.202 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:53:29.202 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-25T20:53:29.202 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-25T20:53:29.340 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:53:29.340 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:53:29.340 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:53:29.340 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2023-12-25T20:53:29.340 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 18, 2023-12-25T20:53:29.340 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:53:29.340 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:53:29.340 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:53:29.341 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:53:29.341 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:29.341 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2023-12-25T20:53:29.341 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:53:29.341 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:53:29.341 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:53:29.341 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:29.341 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:29.341 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:29.341 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:29.341 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:29.341 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:29.341 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:29.341 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:29.342 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:29.342 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:53:29.342 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:53:29.342 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:29.342 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:29.342 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:29.342 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:29.342 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:29.342 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:29.342 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:29.342 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:29.342 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:29.342 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.342 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:53:29.342 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:53:29.343 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:53:29.343 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:53:29.343 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:53:29.343 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:53:29.343 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:53:29.343 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:53:29.343 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:53:29.343 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:53:29.343 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:53:29.343 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:53:29.343 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:53:29.343 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:53:29.343 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:53:29.343 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:53:29.343 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:53:29.344 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:29.344 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:29.344 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:29.344 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:29.344 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:29.344 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:29.344 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:29.344 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:29.344 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:29.344 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:53:29.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.344 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:53:29.344 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.344 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:53:29.345 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:53:29.345 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:29.345 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:29.345 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.345 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:29.345 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:53:29.345 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:29.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.345 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.345 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:29.346 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:53:29.346 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:29.346 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:29.346 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:29.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.346 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:53:29.346 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:53:29.346 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:29.346 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:29.346 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:29.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.346 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.347 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:53:29.347 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:53:29.347 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:29.347 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:29.347 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.347 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:29.347 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:53:29.347 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:29.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.347 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.347 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:29.347 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:53:29.347 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:29.347 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:29.347 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:29.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.348 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:53:29.348 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:53:29.348 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:29.348 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:29.348 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:29.348 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.348 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.348 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:53:29.348 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:53:29.348 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:29.348 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:29.348 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.348 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:29.348 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:53:29.348 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:29.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.349 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.349 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:29.349 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:53:29.349 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:29.349 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:29.349 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:29.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.349 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:53:29.349 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:53:29.349 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:29.349 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:29.349 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:29.349 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:29.349 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:29.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.350 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:53:29.350 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:53:29.350 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.350 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:29.350 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:29.350 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:53:29.350 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:29.350 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:29.350 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:53:29.350 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:29.350 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:29.350 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:29.350 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:53:29.351 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:29.351 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:29.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:29.351 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:53:29.351 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:53:29.352 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:53:29.352 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2023-12-25T20:53:29.352 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2023-12-25T20:53:29.352 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2023-12-25T20:53:29.352 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-12-25T20:53:29.353 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-25T20:53:29.353 INFO:teuthology.orchestra.run:waiting for 300 2023-12-25T20:53:29.364 INFO:tasks.ceph.mon.a:Stopped 2023-12-25T20:53:29.365 INFO:tasks.mon_thrash:Sending STOP to mon b 2023-12-25T20:53:29.365 INFO:tasks.ceph.mon.b:Sent signal 19 2023-12-25T20:53:29.365 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2023-12-25T20:53:30.001 INFO:tasks.workunit.client.0.smithi191.stderr:Note: switching to 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2'. 2023-12-25T20:53:30.001 INFO:tasks.workunit.client.0.smithi191.stderr: 2023-12-25T20:53:30.001 INFO:tasks.workunit.client.0.smithi191.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-25T20:53:30.001 INFO:tasks.workunit.client.0.smithi191.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-25T20:53:30.001 INFO:tasks.workunit.client.0.smithi191.stderr:state without impacting any branches by switching back to a branch. 2023-12-25T20:53:30.001 INFO:tasks.workunit.client.0.smithi191.stderr: 2023-12-25T20:53:30.001 INFO:tasks.workunit.client.0.smithi191.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-25T20:53:30.001 INFO:tasks.workunit.client.0.smithi191.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-25T20:53:30.001 INFO:tasks.workunit.client.0.smithi191.stderr: 2023-12-25T20:53:30.001 INFO:tasks.workunit.client.0.smithi191.stderr: git switch -c 2023-12-25T20:53:30.002 INFO:tasks.workunit.client.0.smithi191.stderr: 2023-12-25T20:53:30.002 INFO:tasks.workunit.client.0.smithi191.stderr:Or undo this operation with: 2023-12-25T20:53:30.002 INFO:tasks.workunit.client.0.smithi191.stderr: 2023-12-25T20:53:30.002 INFO:tasks.workunit.client.0.smithi191.stderr: git switch - 2023-12-25T20:53:30.002 INFO:tasks.workunit.client.0.smithi191.stderr: 2023-12-25T20:53:30.002 INFO:tasks.workunit.client.0.smithi191.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-25T20:53:30.002 INFO:tasks.workunit.client.0.smithi191.stderr: 2023-12-25T20:53:30.002 INFO:tasks.workunit.client.0.smithi191.stderr:HEAD is now at b14a0a106b8 Merge branch 'wip-pacific-rhel-8.6' of https://github.com/cbodley/ceph into wip-yuri5-testing-2023-12-15-0747-pacific 2023-12-25T20:53:30.006 DEBUG:teuthology.orchestra.run.smithi191:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-25T20:53:30.033 INFO:tasks.workunit.client.0.smithi191.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-25T20:53:30.035 INFO:tasks.workunit.client.0.smithi191.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-25T20:53:30.035 INFO:tasks.workunit.client.0.smithi191.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-25T20:53:30.567 INFO:tasks.workunit.client.0.smithi191.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-25T20:53:30.620 INFO:tasks.workunit.client.0.smithi191.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-25T20:53:30.657 INFO:tasks.workunit.client.0.smithi191.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-25T20:53:30.661 INFO:tasks.workunit.client.0.smithi191.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-25T20:53:30.661 INFO:tasks.workunit.client.0.smithi191.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-25T20:53:30.695 INFO:tasks.workunit.client.0.smithi191.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-25T20:53:30.697 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-25T20:53:30.698 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-25T20:53:30.755 INFO:tasks.workunit:Running workunits matching mon/pool_ops.sh on client.0... 2023-12-25T20:53:30.756 INFO:tasks.workunit:Running workunit mon/pool_ops.sh... 2023-12-25T20:53:30.756 DEBUG:teuthology.orchestra.run.smithi191:workunit test mon/pool_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/pool_ops.sh 2023-12-25T20:53:30.817 INFO:tasks.workunit.client.0.smithi191.stderr:+ TEST_POOL=testpool1234 2023-12-25T20:53:30.818 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool create testpool1234 8 --autoscale-mode off 2023-12-25T20:53:44.366 INFO:tasks.mon_thrash:Sending CONT to mon b 2023-12-25T20:53:44.367 INFO:tasks.ceph.mon.b:Sent signal 18 2023-12-25T20:53:44.367 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-12-25T20:53:44.367 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:53:49.597 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'testpool1234' already exists 2023-12-25T20:53:49.607 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool set testpool1234 pg_num_min 2 2023-12-25T20:53:50.745 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":20,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:53:50.759 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-12-25T20:53:50.759 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-12-25T20:53:50.759 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-25T20:53:50.897 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:53:50.897 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:53:50.898 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:53:50.898 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2023-12-25T20:53:50.898 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 20, 2023-12-25T20:53:50.898 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:53:50.898 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:53:50.898 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:53:50.898 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:50.898 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2023-12-25T20:53:50.898 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:53:50.898 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:53:50.898 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:53:50.898 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:50.898 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:50.898 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:50.898 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:50.899 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:50.899 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:50.899 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:50.899 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:50.899 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:50.899 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:53:50.899 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:53:50.899 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:50.899 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:50.899 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:50.899 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:50.899 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:50.899 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:50.899 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:50.899 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:50.900 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:50.900 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:50.900 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:53:50.900 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:53:50.900 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:53:50.900 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:53:50.900 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:53:50.900 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:53:50.900 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:53:50.900 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:53:50.900 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:53:50.900 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:53:50.900 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:53:50.900 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:53:50.901 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:53:50.901 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:53:50.901 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:53:50.901 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:53:50.901 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:53:50.901 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:50.901 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:50.901 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:50.901 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:50.901 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:50.901 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:50.901 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:50.901 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:50.902 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:50.902 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:53:50.902 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:50.902 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:53:50.902 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:50.902 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:53:50.902 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:53:50.902 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:50.902 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:50.902 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:50.902 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:50.902 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:53:50.902 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:50.902 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:50.902 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:50.903 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:50.903 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:53:50.903 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:50.903 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:50.903 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:50.903 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:50.903 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:53:50.903 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:53:50.903 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:50.903 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:50.903 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:50.903 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:50.903 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:50.903 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:53:50.903 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:53:50.904 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:50.904 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:50.904 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:50.904 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:50.904 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:53:50.904 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:50.904 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:50.904 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:50.904 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:50.904 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:53:50.904 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:50.904 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:50.904 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:50.904 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:50.904 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:53:50.905 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:53:50.905 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:50.905 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:50.905 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:50.905 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:50.905 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:50.905 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:53:50.905 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:53:50.905 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:50.906 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:50.906 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:50.906 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:50.906 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:53:50.906 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:50.906 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:50.906 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:50.906 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:50.906 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:53:50.906 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:50.906 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:50.906 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:50.906 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:50.906 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:53:50.906 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2023-12-25T20:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2023-12-25T20:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2023-12-25T20:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:50.909 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:50.909 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:50.909 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:53:50.909 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:50.909 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:50.909 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:50.909 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:53:50.909 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:53:50.910 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-25T20:53:50.910 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-12-25T20:53:50.910 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-25T20:53:51.231 INFO:teuthology.orchestra.run.smithi153.stderr:2023-12-25T20:53:51.231+0000 7fba667fc700 0 --2- 172.21.15.153:0/425909749 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba68099e20 0x7fba68098860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-12-25T20:53:51.248 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:53:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:53:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:53:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2023-12-25T20:53:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 20, 2023-12-25T20:53:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:53:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:53:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:53:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2023-12-25T20:53:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:53:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:53:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:53:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:53:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:53:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:51.251 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:51.251 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:51.251 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:51.251 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:51.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:51.251 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:53:51.251 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:53:51.251 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:53:51.251 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:53:51.251 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:53:51.251 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:53:51.251 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:53:51.251 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:53:51.251 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:53:51.251 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:53:51.251 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:53:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:53:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:53:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:53:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:53:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:53:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:53:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:53:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:53:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:53:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:53:51.253 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:53:51.253 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:53:51.253 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:53:51.253 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:53:51.253 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:51.253 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:53:51.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:51.253 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:53:51.253 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:51.254 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:53:51.254 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:53:51.254 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:51.254 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:51.254 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:51.254 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:51.254 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:53:51.254 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:51.254 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:51.254 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:51.254 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:51.254 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:53:51.254 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:51.254 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:51.254 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:51.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:51.255 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:53:51.255 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:53:51.255 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:51.255 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:51.255 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:51.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:51.255 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:51.255 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:53:51.255 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:53:51.255 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:51.255 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:51.255 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:51.255 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:51.255 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:53:51.256 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:51.256 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:51.256 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:51.256 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:51.256 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:53:51.256 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:51.256 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:51.256 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:51.256 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:51.256 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:53:51.256 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:53:51.256 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:51.256 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:51.256 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:51.256 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:51.257 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:51.257 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:53:51.257 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:53:51.257 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:53:51.257 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:53:51.257 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:51.257 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:53:51.257 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:53:51.257 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:51.257 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:51.257 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:51.257 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:53:51.257 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:53:51.257 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:53:51.257 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:51.258 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:51.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:51.258 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:53:51.258 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:53:51.258 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:53:51.258 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:53:51.258 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:53:51.258 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:51.258 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:51.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:51.258 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:53:51.258 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:53:51.258 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:51.258 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:51.258 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:51.258 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:53:51.259 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:51.259 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:51.259 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:53:51.259 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:51.259 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:51.259 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:51.259 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:53:51.259 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:51.259 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:53:51.259 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:53:51.259 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:53:51.259 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:53:51.259 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:53:51.259 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2023-12-25T20:53:51.259 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:53:51.260 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:53:51.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:53:51.260 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:53:51.260 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:53:51.261 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-25T20:53:51.261 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-12-25T20:53:52.497 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 3 pg_num_min to 2 2023-12-25T20:53:52.509 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool get testpool1234 pg_num_min 2023-12-25T20:53:52.509 INFO:tasks.workunit.client.0.smithi191.stderr:+ grep 2 2023-12-25T20:53:52.898 INFO:tasks.workunit.client.0.smithi191.stdout:pg_num_min: 2 2023-12-25T20:53:52.899 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool set testpool1234 pg_num_max 33 2023-12-25T20:53:54.510 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 3 pg_num_max to 33 2023-12-25T20:53:54.520 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool get testpool1234 pg_num_max 2023-12-25T20:53:54.520 INFO:tasks.workunit.client.0.smithi191.stderr:+ grep 33 2023-12-25T20:53:54.913 INFO:tasks.workunit.client.0.smithi191.stdout:pg_num_max: 33 2023-12-25T20:53:54.913 INFO:tasks.workunit.client.0.smithi191.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_min 9 2023-12-25T20:53:54.913 INFO:tasks.workunit.client.0.smithi191.stderr:+ set -x 2023-12-25T20:53:54.913 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool set testpool1234 pg_num_min 9 2023-12-25T20:53:56.011 INFO:tasks.workunit.client.0.smithi191.stderr:Error EINVAL: specified pg_num_min 9 > pg_num 8 2023-12-25T20:53:56.013 INFO:tasks.workunit.client.0.smithi191.stderr:+ return 0 2023-12-25T20:53:56.014 INFO:tasks.workunit.client.0.smithi191.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_max 7 2023-12-25T20:53:56.014 INFO:tasks.workunit.client.0.smithi191.stderr:+ set -x 2023-12-25T20:53:56.014 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool set testpool1234 pg_num_max 7 2023-12-25T20:53:56.292 INFO:tasks.workunit.client.0.smithi191.stderr:Error EINVAL: specified pg_num_max 7 < pg_num 8 2023-12-25T20:53:56.293 INFO:tasks.workunit.client.0.smithi191.stderr:+ return 0 2023-12-25T20:53:56.293 INFO:tasks.workunit.client.0.smithi191.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 1 2023-12-25T20:53:56.294 INFO:tasks.workunit.client.0.smithi191.stderr:+ set -x 2023-12-25T20:53:56.294 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool set testpool1234 pg_num 1 2023-12-25T20:53:57.181 INFO:tasks.workunit.client.0.smithi191.stderr:Error EINVAL: specified pg_num 1 < pg_num_min 2 2023-12-25T20:53:57.182 INFO:tasks.workunit.client.0.smithi191.stderr:+ return 0 2023-12-25T20:53:57.183 INFO:tasks.workunit.client.0.smithi191.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 44 2023-12-25T20:53:57.183 INFO:tasks.workunit.client.0.smithi191.stderr:+ set -x 2023-12-25T20:53:57.183 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool set testpool1234 pg_num 44 2023-12-25T20:53:57.453 INFO:tasks.workunit.client.0.smithi191.stderr:Error EINVAL: specified pg_num 44 < pg_num_max 33 2023-12-25T20:53:57.454 INFO:tasks.workunit.client.0.smithi191.stderr:+ return 0 2023-12-25T20:53:57.455 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool set testpool1234 pg_num_min 0 2023-12-25T20:54:00.381 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 3 pg_num_min to 0 2023-12-25T20:54:00.393 INFO:tasks.workunit.client.0.smithi191.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_min 2023-12-25T20:54:00.393 INFO:tasks.workunit.client.0.smithi191.stderr:+ set -x 2023-12-25T20:54:00.393 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool get testpool1234 pg_num_min 2023-12-25T20:54:00.668 INFO:tasks.workunit.client.0.smithi191.stderr:Error ENOENT: option 'pg_num_min' is not set on pool 'testpool1234' 2023-12-25T20:54:00.669 INFO:tasks.workunit.client.0.smithi191.stderr:+ return 0 2023-12-25T20:54:00.670 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool set testpool1234 pg_num_max 0 2023-12-25T20:54:03.542 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 3 pg_num_max to 0 2023-12-25T20:54:03.554 INFO:tasks.workunit.client.0.smithi191.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_max 2023-12-25T20:54:03.554 INFO:tasks.workunit.client.0.smithi191.stderr:+ set -x 2023-12-25T20:54:03.554 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool get testpool1234 pg_num_max 2023-12-25T20:54:04.659 INFO:tasks.workunit.client.0.smithi191.stderr:Error ENOENT: option 'pg_num_max' is not set on pool 'testpool1234' 2023-12-25T20:54:04.661 INFO:tasks.workunit.client.0.smithi191.stderr:+ return 0 2023-12-25T20:54:04.661 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool delete testpool1234 testpool1234 --yes-i-really-really-mean-it 2023-12-25T20:54:05.668 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'testpool1234' does not exist 2023-12-25T20:54:05.678 INFO:tasks.workunit.client.0.smithi191.stderr:+ expect_false ceph osd pool create foo 123 123 invalid foo-profile foo-ruleset 2023-12-25T20:54:05.678 INFO:tasks.workunit.client.0.smithi191.stderr:+ set -x 2023-12-25T20:54:05.678 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool create foo 123 123 invalid foo-profile foo-ruleset 2023-12-25T20:54:05.953 INFO:tasks.workunit.client.0.smithi191.stderr:foo-ruleset not valid: foo-ruleset doesn't represent a float 2023-12-25T20:54:05.953 INFO:tasks.workunit.client.0.smithi191.stderr:Invalid command: unused arguments: ['foo-ruleset'] 2023-12-25T20:54:05.953 INFO:tasks.workunit.client.0.smithi191.stderr:osd pool create [] [] [replicated|erasure] [] [] [] [] [] [] [on|off|warn] [--bulk] [] [] : create pool 2023-12-25T20:54:05.953 INFO:tasks.workunit.client.0.smithi191.stderr:Error EINVAL: invalid command 2023-12-25T20:54:05.955 INFO:tasks.workunit.client.0.smithi191.stderr:+ return 0 2023-12-25T20:54:05.955 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool create foo 123 123 replicated 2023-12-25T20:54:06.679 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'foo' already exists 2023-12-25T20:54:06.688 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool create fooo 123 123 erasure default 2023-12-25T20:54:09.707 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'fooo' already exists 2023-12-25T20:54:09.716 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool create foooo 123 2023-12-25T20:54:10.741 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'foooo' already exists 2023-12-25T20:54:10.750 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool create foo 123 2023-12-25T20:54:11.124 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'foo' already exists 2023-12-25T20:54:11.134 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool set foo size 1 --yes-i-really-mean-it 2023-12-25T20:54:11.262 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-12-25T20:54:11.262 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2023-12-25T20:54:11.262 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-25T20:54:11.262 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-12-25T20:54:11.265 INFO:tasks.ceph.mon.a:Started 2023-12-25T20:54:11.265 INFO:tasks.mon_thrash:Sending STOP to mon b 2023-12-25T20:54:11.266 INFO:tasks.ceph.mon.b:Sent signal 19 2023-12-25T20:54:11.266 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2023-12-25T20:54:26.266 INFO:tasks.mon_thrash:Sending CONT to mon b 2023-12-25T20:54:26.267 INFO:tasks.ceph.mon.b:Sent signal 18 2023-12-25T20:54:26.267 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-25T20:54:26.267 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:54:37.254 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:54:37.268 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-25T20:54:37.268 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-25T20:54:37.268 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-25T20:54:37.405 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:54:37.405 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:54:37.405 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:54:37.405 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2023-12-25T20:54:37.406 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 30, 2023-12-25T20:54:37.406 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:54:37.406 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:54:37.406 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:54:37.406 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:54:37.406 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:37.406 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2023-12-25T20:54:37.406 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:54:37.406 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:54:37.406 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:54:37.406 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:37.406 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:37.406 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:37.406 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:37.407 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:37.407 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:37.407 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:37.407 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:37.407 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:37.407 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:54:37.407 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:54:37.407 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:37.407 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:37.407 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:37.407 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:37.407 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:37.407 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:37.407 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:37.407 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:37.408 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:37.408 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.408 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:54:37.408 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:54:37.408 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:54:37.408 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:54:37.408 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:54:37.408 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:54:37.408 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:54:37.408 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:54:37.408 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:54:37.408 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:54:37.408 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:54:37.408 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:54:37.409 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:54:37.409 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:54:37.409 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:54:37.409 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:54:37.409 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:54:37.409 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:37.409 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:37.409 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:37.409 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:37.409 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:37.409 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:37.409 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:37.409 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:37.409 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:37.410 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:54:37.410 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.410 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:54:37.410 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.410 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:54:37.410 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:54:37.410 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:37.410 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:37.410 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.410 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:37.410 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:54:37.410 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:37.410 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.410 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.410 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:37.411 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:54:37.411 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:37.411 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:37.411 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:37.411 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.411 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:54:37.411 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:54:37.411 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:37.411 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:37.411 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:37.411 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.411 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.411 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:54:37.411 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:54:37.411 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:37.411 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:37.412 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.412 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:37.412 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:54:37.412 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:37.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.412 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.412 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:37.412 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:54:37.412 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:37.413 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:37.413 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:37.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.413 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:54:37.413 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:54:37.413 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:37.413 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:37.413 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:37.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.413 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.413 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:54:37.413 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:54:37.413 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:37.414 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:37.414 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.414 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:37.414 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:54:37.414 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:37.414 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.414 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.414 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:37.414 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:54:37.414 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:37.414 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:37.415 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:37.415 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.415 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:54:37.415 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:54:37.415 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:37.415 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:37.415 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:37.415 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:37.415 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:37.415 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.415 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:54:37.415 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:54:37.415 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.415 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:37.415 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:37.416 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:54:37.416 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:37.416 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:37.416 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:54:37.416 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.416 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:37.416 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:37.416 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2023-12-25T20:54:37.416 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:37.416 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:37.416 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.416 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:54:37.416 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:54:37.417 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:54:37.417 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-25T20:54:37.418 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-25T20:54:37.554 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:54:37.554 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:54:37.554 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:54:37.555 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2023-12-25T20:54:37.555 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 30, 2023-12-25T20:54:37.555 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:54:37.555 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:54:37.555 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:54:37.555 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:54:37.555 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:37.555 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2023-12-25T20:54:37.555 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:54:37.555 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:54:37.555 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:54:37.555 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:37.555 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:37.555 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:37.555 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:37.556 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:37.556 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:37.556 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:37.556 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:37.556 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:37.556 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:54:37.556 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:54:37.556 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:37.556 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:37.556 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:37.556 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:37.556 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:37.556 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:37.556 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:37.557 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:37.557 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:37.557 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.557 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:54:37.557 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:54:37.557 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:54:37.557 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:54:37.557 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:54:37.557 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:54:37.557 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:54:37.557 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:54:37.557 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:54:37.557 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:54:37.558 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:54:37.558 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:54:37.558 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:54:37.558 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:54:37.558 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:54:37.558 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:54:37.558 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:54:37.558 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:37.558 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:37.558 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:37.558 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:37.559 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:37.559 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:37.559 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:37.559 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:37.559 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:37.559 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:54:37.559 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.559 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:54:37.559 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.559 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:54:37.559 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:54:37.559 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:37.560 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:37.560 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.560 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:37.560 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:54:37.560 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:37.560 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.560 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.561 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:37.561 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:54:37.561 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:37.561 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:37.561 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:37.561 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.561 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:54:37.561 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:54:37.561 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:37.561 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:37.561 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:37.561 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.561 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.562 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:54:37.562 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:54:37.562 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:37.562 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:37.562 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.562 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:37.562 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:54:37.562 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:37.562 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.562 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.562 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:37.562 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:54:37.562 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:37.562 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:37.563 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:37.563 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.563 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:54:37.563 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:54:37.563 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:37.563 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:37.563 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:37.563 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.563 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.563 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:54:37.563 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:54:37.563 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:37.563 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:37.563 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.563 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:37.563 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:54:37.564 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:37.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.564 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.564 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:37.564 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:54:37.564 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:37.564 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:37.564 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:37.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.564 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:54:37.564 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:54:37.564 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:37.564 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:37.564 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:37.565 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:37.565 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:37.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.566 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:54:37.566 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:54:37.566 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.566 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:37.566 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:37.566 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:54:37.566 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:37.567 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:37.567 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:54:37.567 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.567 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:37.567 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:37.567 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2023-12-25T20:54:37.567 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:37.567 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:37.567 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:54:37.567 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.567 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:37.567 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:37.567 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2023-12-25T20:54:37.568 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:37.568 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:37.568 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2023-12-25T20:54:37.568 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.568 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:37.568 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:37.568 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:54:37.568 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:37.568 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:37.568 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.568 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:54:37.568 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:54:37.570 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:54:37.570 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-25T20:54:37.570 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-25T20:54:37.709 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:54:37.710 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:54:37.710 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:54:37.710 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2023-12-25T20:54:37.710 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 30, 2023-12-25T20:54:37.710 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:54:37.710 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:54:37.710 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:54:37.710 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:54:37.710 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:37.710 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2023-12-25T20:54:37.710 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:54:37.710 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:54:37.710 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:54:37.710 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:37.710 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:37.711 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:37.711 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:37.711 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:37.711 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:37.711 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:37.711 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:37.711 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:37.711 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:54:37.711 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:54:37.711 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:37.711 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:37.711 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:37.711 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:37.711 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:37.711 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:37.712 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:37.712 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:37.712 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:37.712 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.712 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:54:37.712 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:54:37.712 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:54:37.712 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:54:37.712 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:54:37.712 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:54:37.712 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:54:37.712 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:54:37.712 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:54:37.713 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:54:37.713 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:54:37.713 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:54:37.713 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:54:37.713 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:54:37.713 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:54:37.713 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:54:37.713 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:54:37.713 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:37.713 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:37.713 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:37.713 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:37.713 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:37.713 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:37.714 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:37.714 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:37.714 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:37.714 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:54:37.714 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.714 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:54:37.714 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.714 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:54:37.714 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:54:37.714 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:37.714 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:37.714 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.714 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:37.714 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:54:37.715 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:37.715 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.715 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.715 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:37.715 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:54:37.715 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:37.715 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:37.715 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:37.715 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.715 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:54:37.715 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:54:37.715 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:37.715 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:37.715 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:37.715 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.716 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.716 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:54:37.716 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:54:37.716 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:37.716 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:37.716 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.716 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:37.716 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:54:37.716 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:37.716 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.716 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.716 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:37.716 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:54:37.717 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:37.717 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:37.717 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:37.717 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.717 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:54:37.717 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:54:37.717 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:37.717 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:37.717 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:37.717 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.717 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.717 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:54:37.717 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:54:37.717 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:37.718 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:37.718 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.718 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:37.718 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:54:37.718 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:37.718 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.718 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.718 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:37.718 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:54:37.718 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:37.718 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:37.718 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:37.718 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.719 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:54:37.719 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:54:37.719 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:37.719 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:37.719 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:37.719 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:37.719 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:37.719 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.719 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:54:37.719 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:54:37.720 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.720 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:37.720 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:37.720 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:54:37.720 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:37.720 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:37.720 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:54:37.720 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.720 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:37.720 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:37.720 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:54:37.720 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:37.720 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:37.720 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:54:37.720 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:37.721 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:37.721 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:37.721 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2023-12-25T20:54:37.721 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:37.721 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:37.721 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:37.721 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:54:37.721 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:54:37.722 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:54:37.722 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-12-25T20:54:37.722 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-12-25T20:54:37.812 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 4 size to 1 2023-12-25T20:54:37.830 INFO:tasks.workunit.client.0.smithi191.stderr:+ expect_config_value foo min_size 1 2023-12-25T20:54:37.830 INFO:tasks.workunit.client.0.smithi191.stderr:+ local pool_name config_opt expected_val val 2023-12-25T20:54:37.830 INFO:tasks.workunit.client.0.smithi191.stderr:+ pool_name=foo 2023-12-25T20:54:37.830 INFO:tasks.workunit.client.0.smithi191.stderr:+ config_opt=min_size 2023-12-25T20:54:37.830 INFO:tasks.workunit.client.0.smithi191.stderr:+ expected_val=1 2023-12-25T20:54:37.830 INFO:tasks.workunit.client.0.smithi191.stderr:++ get_config_value_or_die foo min_size 2023-12-25T20:54:37.830 INFO:tasks.workunit.client.0.smithi191.stderr:++ local pool_name config_opt raw val 2023-12-25T20:54:37.830 INFO:tasks.workunit.client.0.smithi191.stderr:++ pool_name=foo 2023-12-25T20:54:37.831 INFO:tasks.workunit.client.0.smithi191.stderr:++ config_opt=min_size 2023-12-25T20:54:37.831 INFO:tasks.workunit.client.0.smithi191.stderr:+++ ceph osd pool get foo min_size 2023-12-25T20:54:38.098 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:54:38.097+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:54:38.098 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:54:38.097+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=10,kv=5,logm=72} crc {auth=3421424784,config=3140514009,health=1388609981,kv=731544943,logm=3460139412}) 2023-12-25T20:54:38.098 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:54:38.097+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=11,kv=6,logm=70} crc {auth=3421424784,config=3140514009,health=4237594849,kv=661943861,logm=3335398588}) 2023-12-25T20:54:38.098 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:54:38.097+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:54:38.098 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:54:38.097+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=10,kv=5,logm=72} crc {auth=3421424784,config=3140514009,health=1388609981,kv=731544943,logm=3460139412}) 2023-12-25T20:54:38.099 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:54:38.097+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=11,config=2,health=12,kv=5,logm=70} crc {auth=399307255,config=3140514009,health=1148745368,kv=731544943,logm=2757314239}) 2023-12-25T20:54:38.100 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:54:38.099+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:54:38.100 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:54:38.099+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=44,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=18} crc {logm=4035314086,mdsmap=1531151339,mgr=3903938553,mgr_command_descs=1448462394,mgr_metadata=1482708640,mgrstat=4045777317,mon_config_key=2934451455,monmap=3296645973,osd_metadata=1683759159,osd_pg_creating=4226738956,osdmap=3419257815}) 2023-12-25T20:54:38.100 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:54:38.099+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=40,mdsmap=2,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=3,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=25} crc {logm=3432848540,mdsmap=3585761405,mgr=1153494972,mgr_command_descs=1448462394,mgr_metadata=1482708640,mgrstat=4045777317,mon_config_key=2934451455,monmap=3474379421,osd_metadata=1683759159,osd_pg_creating=4226738956,osdmap=3655396198}) 2023-12-25T20:54:38.100 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:54:38.099+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:54:38.100 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:54:38.099+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=44,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=18} crc {logm=4035314086,mdsmap=1531151339,mgr=3903938553,mgr_command_descs=1448462394,mgr_metadata=1482708640,mgrstat=4045777317,mon_config_key=2934451455,monmap=3296645973,osd_metadata=1683759159,osd_pg_creating=4226738956,osdmap=3419257815}) 2023-12-25T20:54:38.100 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:54:38.099+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=45,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=19} crc {logm=4224940069,mdsmap=1531151339,mgr=2949565714,mgr_command_descs=1448462394,mgr_metadata=1482708640,mgrstat=3312441386,mon_config_key=2934451455,monmap=3296645973,osd_metadata=1683759159,osd_pg_creating=4226738956,osdmap=63408424}) 2023-12-25T20:54:38.101 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:54:38.101+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:54:38.101 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:54:38.101+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=52} crc {osdmap=4090909556}) 2023-12-25T20:54:38.102 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:54:38.101+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=48} crc {osdmap=2259990253}) 2023-12-25T20:54:38.102 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:54:38.101+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:54:38.102 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:54:38.101+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=52} crc {osdmap=4090909556}) 2023-12-25T20:54:38.102 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:54:38.101+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=49} crc {osdmap=2454016333}) 2023-12-25T20:54:38.112 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-12-25T20:54:38.220 INFO:tasks.workunit.client.0.smithi191.stderr:++ raw='min_size: 1' 2023-12-25T20:54:38.221 INFO:tasks.workunit.client.0.smithi191.stderr:++ [[ 0 -ne 0 ]] 2023-12-25T20:54:38.221 INFO:tasks.workunit.client.0.smithi191.stderr:+++ echo min_size: 1 2023-12-25T20:54:38.221 INFO:tasks.workunit.client.0.smithi191.stderr:+++ sed -e 's/[{} "]//g' 2023-12-25T20:54:38.222 INFO:tasks.workunit.client.0.smithi191.stderr:++ raw=min_size:1 2023-12-25T20:54:38.222 INFO:tasks.workunit.client.0.smithi191.stderr:+++ echo min_size:1 2023-12-25T20:54:38.222 INFO:tasks.workunit.client.0.smithi191.stderr:+++ cut -f2 -d: 2023-12-25T20:54:38.223 INFO:tasks.workunit.client.0.smithi191.stderr:++ val=1 2023-12-25T20:54:38.223 INFO:tasks.workunit.client.0.smithi191.stderr:++ echo 1 2023-12-25T20:54:38.223 INFO:tasks.workunit.client.0.smithi191.stderr:++ return 0 2023-12-25T20:54:38.223 INFO:tasks.workunit.client.0.smithi191.stderr:+ val=1 2023-12-25T20:54:38.223 INFO:tasks.workunit.client.0.smithi191.stderr:+ [[ 1 != \1 ]] 2023-12-25T20:54:38.223 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool set foo size 4 2023-12-25T20:54:39.113 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-25T20:54:39.114 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:54:39.486 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:54:39.499 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-25T20:54:39.499 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-12-25T20:54:39.499 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-25T20:54:39.499 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-25T20:54:39.639 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:54:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:54:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:54:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2023-12-25T20:54:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 30, 2023-12-25T20:54:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:54:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:54:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:54:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:54:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2023-12-25T20:54:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:54:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:54:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:54:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:54:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:54:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:54:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:54:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:54:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:54:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:54:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:54:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:54:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:54:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:54:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:54:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:54:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:54:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:54:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:54:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:54:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:54:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:54:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:54:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:54:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:54:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:54:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:54:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:54:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:54:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:54:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:54:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:54:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:54:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:54:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:54:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:54:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:54:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:54:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:54:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:54:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:54:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:54:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:54:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:54:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:54:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:54:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2023-12-25T20:54:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:54:39.649 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:54:39.650 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:54:39.650 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-25T20:54:39.651 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-25T20:54:39.787 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:54:39.787 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:54:39.787 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:54:39.787 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2023-12-25T20:54:39.787 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 30, 2023-12-25T20:54:39.787 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:54:39.788 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:54:39.788 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:54:39.788 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:54:39.788 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:39.788 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2023-12-25T20:54:39.788 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:54:39.788 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:54:39.788 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:54:39.788 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:39.788 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:39.788 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:39.788 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:39.788 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:39.788 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:39.789 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:39.789 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:39.789 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:39.789 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:54:39.789 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:54:39.789 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:39.789 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:39.789 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:39.789 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:39.789 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:39.789 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:39.789 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:39.789 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:39.789 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:39.790 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.790 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:54:39.790 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:54:39.790 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:54:39.790 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:54:39.790 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:54:39.790 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:54:39.790 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:54:39.790 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:54:39.790 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:54:39.790 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:54:39.790 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:54:39.790 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:54:39.790 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:54:39.790 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:54:39.791 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:54:39.791 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:54:39.791 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:54:39.791 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:39.791 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:39.791 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:39.791 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:39.791 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:39.792 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:39.792 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:39.792 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:39.792 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:39.792 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:54:39.792 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.792 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:54:39.792 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.792 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:54:39.792 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:54:39.792 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:39.792 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:39.792 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.792 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:39.793 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:54:39.793 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:39.793 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.793 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.793 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:39.793 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:54:39.793 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:39.793 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:39.793 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:39.793 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.793 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:54:39.793 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:54:39.793 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:39.793 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:39.794 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:39.794 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.794 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.794 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:54:39.794 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:54:39.794 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:39.794 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:39.794 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.794 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:39.794 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:54:39.794 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:39.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.795 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.795 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:39.795 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:54:39.795 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:39.795 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:39.795 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:39.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.795 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:54:39.795 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:54:39.795 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:39.795 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:39.795 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:39.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.795 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.796 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:54:39.796 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:54:39.796 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:39.796 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:39.796 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.796 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:39.796 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:54:39.796 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:39.796 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.796 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.796 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:39.796 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:54:39.796 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:39.796 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:39.796 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:39.797 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.797 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:54:39.797 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:54:39.797 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:39.797 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:39.797 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:39.797 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:39.797 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:39.797 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.797 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:54:39.797 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:54:39.797 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.797 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:39.797 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:39.798 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:54:39.798 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:39.798 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:39.798 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:54:39.798 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.798 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:39.798 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:39.798 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2023-12-25T20:54:39.798 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:39.798 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:39.798 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:54:39.798 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.798 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:39.798 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:39.798 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2023-12-25T20:54:39.799 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:39.799 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:39.799 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2023-12-25T20:54:39.799 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.799 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:39.799 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:39.799 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:54:39.799 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:39.799 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:39.799 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.799 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:54:39.799 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:54:39.800 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:54:39.801 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-25T20:54:39.801 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-25T20:54:39.822 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 4 size to 4 2023-12-25T20:54:39.845 INFO:tasks.workunit.client.0.smithi191.stderr:+ expect_config_value foo min_size 2 2023-12-25T20:54:39.846 INFO:tasks.workunit.client.0.smithi191.stderr:+ local pool_name config_opt expected_val val 2023-12-25T20:54:39.846 INFO:tasks.workunit.client.0.smithi191.stderr:+ pool_name=foo 2023-12-25T20:54:39.846 INFO:tasks.workunit.client.0.smithi191.stderr:+ config_opt=min_size 2023-12-25T20:54:39.846 INFO:tasks.workunit.client.0.smithi191.stderr:+ expected_val=2 2023-12-25T20:54:39.847 INFO:tasks.workunit.client.0.smithi191.stderr:++ get_config_value_or_die foo min_size 2023-12-25T20:54:39.847 INFO:tasks.workunit.client.0.smithi191.stderr:++ local pool_name config_opt raw val 2023-12-25T20:54:39.847 INFO:tasks.workunit.client.0.smithi191.stderr:++ pool_name=foo 2023-12-25T20:54:39.847 INFO:tasks.workunit.client.0.smithi191.stderr:++ config_opt=min_size 2023-12-25T20:54:39.847 INFO:tasks.workunit.client.0.smithi191.stderr:+++ ceph osd pool get foo min_size 2023-12-25T20:54:39.982 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:54:39.982 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:54:39.982 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:54:39.982 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2023-12-25T20:54:39.982 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 30, 2023-12-25T20:54:39.982 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:54:39.982 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:54:39.982 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:54:39.982 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:54:39.982 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:39.982 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2023-12-25T20:54:39.982 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:54:39.983 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:54:39.983 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:54:39.983 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:39.983 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:39.983 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:39.983 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:39.983 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:39.983 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:39.983 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:39.983 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:39.983 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:39.983 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:54:39.983 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:54:39.984 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:39.984 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:39.984 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:39.984 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:39.984 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:39.984 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:39.984 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:39.984 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:39.984 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:39.984 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.984 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:54:39.984 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:54:39.984 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:54:39.984 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:54:39.984 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:54:39.985 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:54:39.985 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:54:39.985 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:54:39.985 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:54:39.985 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:54:39.985 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:54:39.985 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:54:39.985 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:54:39.985 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:54:39.985 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:54:39.985 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:54:39.985 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:54:39.985 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:39.985 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:39.986 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:39.986 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:39.986 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:39.986 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:39.986 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:39.986 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:39.986 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:39.986 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:54:39.986 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.986 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:54:39.986 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.986 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:54:39.986 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:54:39.986 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:39.986 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:39.987 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.987 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:39.987 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:54:39.987 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:39.987 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.987 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.987 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:39.987 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:54:39.987 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:39.987 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:39.987 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:39.987 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.987 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:54:39.987 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:54:39.987 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:39.988 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:39.988 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:39.988 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.988 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.988 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:54:39.988 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:54:39.988 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:39.988 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:39.988 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.988 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:39.988 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:54:39.988 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:39.988 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.988 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.988 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:39.989 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:54:39.989 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:39.989 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:39.989 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:39.989 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.989 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:54:39.989 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:54:39.989 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:39.989 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:39.989 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:39.989 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.989 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.989 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:54:39.989 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:54:39.989 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:39.990 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:39.990 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.990 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:39.990 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:54:39.990 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:39.990 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.990 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.990 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:39.990 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:54:39.990 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:39.990 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:39.990 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:39.990 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.990 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:54:39.990 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:54:39.991 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:39.991 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:39.991 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:39.991 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:39.991 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:39.991 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.991 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:54:39.991 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:54:39.991 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.991 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:39.991 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:39.991 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:54:39.991 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:39.991 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:39.991 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:54:39.992 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.992 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:39.992 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:39.992 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:54:39.992 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:39.992 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:39.992 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:54:39.992 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:39.992 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:39.992 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:39.992 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2023-12-25T20:54:39.992 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:39.992 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:39.992 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:39.992 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:54:39.993 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:54:39.994 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:54:39.994 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2023-12-25T20:54:39.994 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-12-25T20:54:39.994 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2023-12-25T20:54:39.994 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2023-12-25T20:54:39.994 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-25T20:54:39.995 INFO:teuthology.orchestra.run:waiting for 300 2023-12-25T20:54:40.005 INFO:tasks.ceph.mon.c:Stopped 2023-12-25T20:54:40.005 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-12-25T20:54:40.006 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:54:40.260 INFO:tasks.workunit.client.0.smithi191.stderr:++ raw='min_size: 2' 2023-12-25T20:54:40.261 INFO:tasks.workunit.client.0.smithi191.stderr:++ [[ 0 -ne 0 ]] 2023-12-25T20:54:40.261 INFO:tasks.workunit.client.0.smithi191.stderr:+++ echo min_size: 2 2023-12-25T20:54:40.261 INFO:tasks.workunit.client.0.smithi191.stderr:+++ sed -e 's/[{} "]//g' 2023-12-25T20:54:40.262 INFO:tasks.workunit.client.0.smithi191.stderr:++ raw=min_size:2 2023-12-25T20:54:40.262 INFO:tasks.workunit.client.0.smithi191.stderr:+++ echo min_size:2 2023-12-25T20:54:40.262 INFO:tasks.workunit.client.0.smithi191.stderr:+++ cut -f2 -d: 2023-12-25T20:54:40.263 INFO:tasks.workunit.client.0.smithi191.stderr:++ val=2 2023-12-25T20:54:40.263 INFO:tasks.workunit.client.0.smithi191.stderr:++ echo 2 2023-12-25T20:54:40.263 INFO:tasks.workunit.client.0.smithi191.stderr:++ return 0 2023-12-25T20:54:40.263 INFO:tasks.workunit.client.0.smithi191.stderr:+ val=2 2023-12-25T20:54:40.263 INFO:tasks.workunit.client.0.smithi191.stderr:+ [[ 2 != \2 ]] 2023-12-25T20:54:40.263 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool set foo size 10 2023-12-25T20:54:40.399 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:54:43.414 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:54:43.780 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:54:46.795 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:54:47.160 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:54:50.175 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:54:55.090 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":34,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:54:55.104 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-12-25T20:54:55.104 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-12-25T20:54:55.104 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-25T20:54:55.244 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:54:55.244 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:54:55.244 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:54:55.245 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2023-12-25T20:54:55.245 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 34, 2023-12-25T20:54:55.245 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:54:55.245 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:54:55.245 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2023-12-25T20:54:55.245 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:55.245 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2023-12-25T20:54:55.245 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:54:55.245 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:54:55.245 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:54:55.245 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:55.245 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:55.246 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:55.246 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:55.246 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:55.246 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:55.246 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:55.246 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:55.246 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:55.246 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:54:55.246 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:54:55.246 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:55.246 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:55.246 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:55.246 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:55.247 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:55.247 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:55.247 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:55.247 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:55.247 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:55.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.247 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:54:55.247 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:54:55.247 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:54:55.247 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:54:55.248 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:54:55.248 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:54:55.248 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:54:55.248 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:54:55.248 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:54:55.248 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:54:55.248 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:54:55.248 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:54:55.248 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:54:55.248 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:54:55.248 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:54:55.248 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:54:55.249 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:54:55.249 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:55.249 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:55.249 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:55.249 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:55.249 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:55.249 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:55.249 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:55.249 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:55.249 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:55.250 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:54:55.250 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.250 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:54:55.250 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.250 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:54:55.250 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:54:55.250 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:55.251 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:55.251 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.251 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:55.251 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:54:55.251 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:55.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.251 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.251 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:55.251 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:54:55.252 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:55.252 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:55.252 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:55.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.252 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:54:55.252 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:54:55.252 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:55.252 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:55.253 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:55.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.253 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.253 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:54:55.253 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:54:55.253 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:55.253 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:55.253 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.254 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:55.254 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:54:55.254 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:55.254 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.254 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.254 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:55.254 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:54:55.254 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:55.255 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:55.255 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:55.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.255 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:54:55.255 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:54:55.255 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:55.255 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:55.255 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:55.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.255 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.255 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:54:55.255 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:54:55.256 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:55.256 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:55.256 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.256 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:55.256 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:54:55.256 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:55.256 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.256 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.256 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:55.257 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:54:55.257 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:55.257 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:55.257 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:55.257 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.257 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:54:55.257 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:54:55.258 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:55.258 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:55.258 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:55.258 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:55.258 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:55.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.258 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:54:55.258 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:54:55.259 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.259 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:55.259 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:55.259 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:54:55.259 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:55.259 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:55.259 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:54:55.259 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.260 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:55.260 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:55.260 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2023-12-25T20:54:55.260 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:55.260 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:55.260 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:54:55.260 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.260 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:55.260 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:55.261 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2023-12-25T20:54:55.261 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:55.261 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:55.261 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.261 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:54:55.261 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:54:55.263 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-25T20:54:55.263 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-12-25T20:54:55.263 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-25T20:54:55.408 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:54:55.408 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:54:55.409 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:54:55.409 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2023-12-25T20:54:55.409 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 34, 2023-12-25T20:54:55.409 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:54:55.409 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:54:55.409 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2023-12-25T20:54:55.410 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:55.410 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2023-12-25T20:54:55.410 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:54:55.410 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:54:55.410 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:54:55.410 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:55.411 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:55.411 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:55.411 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:55.411 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:55.411 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:55.411 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:55.411 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:55.412 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:55.412 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:54:55.412 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:54:55.412 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:55.412 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:55.412 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:55.412 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:55.412 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:55.412 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:55.413 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:55.413 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:55.413 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:55.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.413 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:54:55.413 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:54:55.413 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:54:55.414 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:54:55.414 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:54:55.414 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:54:55.414 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:54:55.414 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:54:55.414 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:54:55.414 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:54:55.414 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:54:55.415 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:54:55.415 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:54:55.415 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:54:55.415 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:54:55.415 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:54:55.415 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:54:55.415 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:54:55.415 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:54:55.416 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:54:55.416 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:54:55.416 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:54:55.416 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:54:55.416 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:54:55.416 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:54:55.416 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:55.416 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:54:55.416 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.416 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:54:55.417 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.417 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:54:55.417 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:54:55.417 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:55.417 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:55.417 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.417 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:55.417 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:54:55.417 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:55.417 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.417 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:54:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:54:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:54:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:54:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:54:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:54:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.420 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:55.420 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:54:55.420 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:55.420 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:55.420 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:55.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.421 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:54:55.421 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:54:55.421 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:55.421 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:55.421 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:55.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.422 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.422 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:54:55.422 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:54:55.422 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:54:55.422 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:54:55.422 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.423 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:54:55.423 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:54:55.423 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:55.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.423 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.423 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:54:55.423 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:54:55.423 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:54:55.424 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:55.424 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:55.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.424 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:54:55.424 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:54:55.425 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:54:55.425 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:54:55.425 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:54:55.425 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:55.425 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:55.425 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.426 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:54:55.426 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:54:55.429 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.429 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:55.429 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:55.429 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:54:55.429 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:55.429 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:54:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2023-12-25T20:54:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:54:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2023-12-25T20:54:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:54:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2023-12-25T20:54:55.431 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:54:55.431 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:54:55.431 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:54:55.431 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:54:55.431 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:54:55.431 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:54:55.431 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:54:55.431 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:54:55.431 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:54:55.433 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-25T20:54:55.433 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-12-25T20:54:56.875 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 4 size to 10 2023-12-25T20:54:56.885 INFO:tasks.workunit.client.0.smithi191.stderr:+ expect_config_value foo min_size 5 2023-12-25T20:54:56.885 INFO:tasks.workunit.client.0.smithi191.stderr:+ local pool_name config_opt expected_val val 2023-12-25T20:54:56.885 INFO:tasks.workunit.client.0.smithi191.stderr:+ pool_name=foo 2023-12-25T20:54:56.885 INFO:tasks.workunit.client.0.smithi191.stderr:+ config_opt=min_size 2023-12-25T20:54:56.885 INFO:tasks.workunit.client.0.smithi191.stderr:+ expected_val=5 2023-12-25T20:54:56.885 INFO:tasks.workunit.client.0.smithi191.stderr:++ get_config_value_or_die foo min_size 2023-12-25T20:54:56.886 INFO:tasks.workunit.client.0.smithi191.stderr:++ local pool_name config_opt raw val 2023-12-25T20:54:56.886 INFO:tasks.workunit.client.0.smithi191.stderr:++ pool_name=foo 2023-12-25T20:54:56.886 INFO:tasks.workunit.client.0.smithi191.stderr:++ config_opt=min_size 2023-12-25T20:54:56.886 INFO:tasks.workunit.client.0.smithi191.stderr:+++ ceph osd pool get foo min_size 2023-12-25T20:54:57.315 INFO:tasks.workunit.client.0.smithi191.stderr:++ raw='min_size: 5' 2023-12-25T20:54:57.316 INFO:tasks.workunit.client.0.smithi191.stderr:++ [[ 0 -ne 0 ]] 2023-12-25T20:54:57.316 INFO:tasks.workunit.client.0.smithi191.stderr:+++ echo min_size: 5 2023-12-25T20:54:57.316 INFO:tasks.workunit.client.0.smithi191.stderr:+++ sed -e 's/[{} "]//g' 2023-12-25T20:54:57.316 INFO:tasks.workunit.client.0.smithi191.stderr:++ raw=min_size:5 2023-12-25T20:54:57.317 INFO:tasks.workunit.client.0.smithi191.stderr:+++ echo min_size:5 2023-12-25T20:54:57.317 INFO:tasks.workunit.client.0.smithi191.stderr:+++ cut -f2 -d: 2023-12-25T20:54:57.318 INFO:tasks.workunit.client.0.smithi191.stderr:++ val=5 2023-12-25T20:54:57.318 INFO:tasks.workunit.client.0.smithi191.stderr:++ echo 5 2023-12-25T20:54:57.318 INFO:tasks.workunit.client.0.smithi191.stderr:++ return 0 2023-12-25T20:54:57.318 INFO:tasks.workunit.client.0.smithi191.stderr:+ val=5 2023-12-25T20:54:57.318 INFO:tasks.workunit.client.0.smithi191.stderr:+ [[ 5 != \5 ]] 2023-12-25T20:54:57.319 INFO:tasks.workunit.client.0.smithi191.stderr:+ expect_false ceph osd pool set foo size 0 2023-12-25T20:54:57.319 INFO:tasks.workunit.client.0.smithi191.stderr:+ set -x 2023-12-25T20:54:57.319 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool set foo size 0 2023-12-25T20:54:57.591 INFO:tasks.workunit.client.0.smithi191.stderr:Error EINVAL: pool size must be between 1 and 10 2023-12-25T20:54:57.593 INFO:tasks.workunit.client.0.smithi191.stderr:+ return 0 2023-12-25T20:54:57.593 INFO:tasks.workunit.client.0.smithi191.stderr:+ expect_false ceph osd pool set foo size 20 2023-12-25T20:54:57.593 INFO:tasks.workunit.client.0.smithi191.stderr:+ set -x 2023-12-25T20:54:57.593 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool set foo size 20 2023-12-25T20:54:57.868 INFO:tasks.workunit.client.0.smithi191.stderr:Error EINVAL: pool size must be between 1 and 10 2023-12-25T20:54:57.870 INFO:tasks.workunit.client.0.smithi191.stderr:+ return 0 2023-12-25T20:54:57.870 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool set foo size 3 2023-12-25T20:55:00.230 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 4 size to 3 2023-12-25T20:55:00.245 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd getcrushmap -o crush 2023-12-25T20:55:00.648 INFO:tasks.workunit.client.0.smithi191.stderr:5 2023-12-25T20:55:00.657 INFO:tasks.workunit.client.0.smithi191.stderr:+ crushtool -d crush -o crush.txt 2023-12-25T20:55:00.669 INFO:tasks.workunit.client.0.smithi191.stderr:+ sed -i 's/max_size 10/max_size 3/' crush.txt 2023-12-25T20:55:00.670 INFO:tasks.workunit.client.0.smithi191.stderr:+ crushtool -c crush.txt -o crush.new 2023-12-25T20:55:00.682 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd setcrushmap -i crush.new 2023-12-25T20:55:01.374 INFO:tasks.workunit.client.0.smithi191.stderr:7 2023-12-25T20:55:01.383 INFO:tasks.workunit.client.0.smithi191.stderr:+ expect_false ceph osd pool set foo size 4 2023-12-25T20:55:01.384 INFO:tasks.workunit.client.0.smithi191.stderr:+ set -x 2023-12-25T20:55:01.384 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool set foo size 4 2023-12-25T20:55:01.655 INFO:tasks.workunit.client.0.smithi191.stderr:Error EINVAL: pool size is bigger than the crush rule max size 2023-12-25T20:55:01.660 INFO:tasks.workunit.client.0.smithi191.stderr:+ return 0 2023-12-25T20:55:01.660 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd setcrushmap -i crush 2023-12-25T20:55:03.395 INFO:tasks.workunit.client.0.smithi191.stderr:9 2023-12-25T20:55:03.406 INFO:tasks.workunit.client.0.smithi191.stderr:+ rm -f crush crush.txt crush.new 2023-12-25T20:55:03.407 INFO:tasks.workunit.client.0.smithi191.stderr:+ expect_false ceph osd pool delete foo 2023-12-25T20:55:03.407 INFO:tasks.workunit.client.0.smithi191.stderr:+ set -x 2023-12-25T20:55:03.407 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool delete foo 2023-12-25T20:55:03.680 INFO:tasks.workunit.client.0.smithi191.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2023-12-25T20:55:03.682 INFO:tasks.workunit.client.0.smithi191.stderr:+ return 0 2023-12-25T20:55:03.682 INFO:tasks.workunit.client.0.smithi191.stderr:+ expect_false ceph osd pool delete foo foo 2023-12-25T20:55:03.682 INFO:tasks.workunit.client.0.smithi191.stderr:+ set -x 2023-12-25T20:55:03.682 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool delete foo foo 2023-12-25T20:55:03.954 INFO:tasks.workunit.client.0.smithi191.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2023-12-25T20:55:03.956 INFO:tasks.workunit.client.0.smithi191.stderr:+ return 0 2023-12-25T20:55:03.956 INFO:tasks.workunit.client.0.smithi191.stderr:+ expect_false ceph osd pool delete foo foo --force 2023-12-25T20:55:03.956 INFO:tasks.workunit.client.0.smithi191.stderr:+ set -x 2023-12-25T20:55:03.956 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool delete foo foo --force 2023-12-25T20:55:04.230 INFO:tasks.workunit.client.0.smithi191.stderr:--force not valid: --force not one of 'true', 'false' 2023-12-25T20:55:04.230 INFO:tasks.workunit.client.0.smithi191.stderr:Invalid command: unused arguments: ['--force'] 2023-12-25T20:55:04.230 INFO:tasks.workunit.client.0.smithi191.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2023-12-25T20:55:04.230 INFO:tasks.workunit.client.0.smithi191.stderr:Error EINVAL: invalid command 2023-12-25T20:55:04.232 INFO:tasks.workunit.client.0.smithi191.stderr:+ return 0 2023-12-25T20:55:04.232 INFO:tasks.workunit.client.0.smithi191.stderr:+ expect_false ceph osd pool delete foo fooo --yes-i-really-mean-it 2023-12-25T20:55:04.232 INFO:tasks.workunit.client.0.smithi191.stderr:+ set -x 2023-12-25T20:55:04.232 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool delete foo fooo --yes-i-really-mean-it 2023-12-25T20:55:04.501 INFO:tasks.workunit.client.0.smithi191.stderr:--yes-i-really-mean-it not valid: --yes-i-really-mean-it not one of 'true', 'false' 2023-12-25T20:55:04.501 INFO:tasks.workunit.client.0.smithi191.stderr:Invalid command: unused arguments: ['--yes-i-really-mean-it'] 2023-12-25T20:55:04.501 INFO:tasks.workunit.client.0.smithi191.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2023-12-25T20:55:04.501 INFO:tasks.workunit.client.0.smithi191.stderr:Error EINVAL: invalid command 2023-12-25T20:55:04.503 INFO:tasks.workunit.client.0.smithi191.stderr:+ return 0 2023-12-25T20:55:04.503 INFO:tasks.workunit.client.0.smithi191.stderr:+ expect_false ceph osd pool delete foo --yes-i-really-mean-it foo 2023-12-25T20:55:04.503 INFO:tasks.workunit.client.0.smithi191.stderr:+ set -x 2023-12-25T20:55:04.504 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool delete foo --yes-i-really-mean-it foo 2023-12-25T20:55:04.773 INFO:tasks.workunit.client.0.smithi191.stderr:foo not valid: foo not one of 'true', 'false' 2023-12-25T20:55:04.774 INFO:tasks.workunit.client.0.smithi191.stderr:Invalid command: unused arguments: ['foo'] 2023-12-25T20:55:04.774 INFO:tasks.workunit.client.0.smithi191.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2023-12-25T20:55:04.774 INFO:tasks.workunit.client.0.smithi191.stderr:Error EINVAL: invalid command 2023-12-25T20:55:04.775 INFO:tasks.workunit.client.0.smithi191.stderr:+ return 0 2023-12-25T20:55:04.775 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool delete foooo foooo --yes-i-really-really-mean-it 2023-12-25T20:55:05.514 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'foooo' does not exist 2023-12-25T20:55:05.522 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2023-12-25T20:55:06.494 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'fooo' does not exist 2023-12-25T20:55:06.503 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2023-12-25T20:55:07.544 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'foo' does not exist 2023-12-25T20:55:07.553 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2023-12-25T20:55:07.934 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'foo' does not exist 2023-12-25T20:55:07.944 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2023-12-25T20:55:08.335 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'fooo' does not exist 2023-12-25T20:55:08.344 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2023-12-25T20:55:09.340 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'fooo' does not exist 2023-12-25T20:55:09.349 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool delete fuggg fuggg --yes-i-really-really-mean-it 2023-12-25T20:55:09.726 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'fuggg' does not exist 2023-12-25T20:55:09.735 INFO:tasks.workunit.client.0.smithi191.stderr:+ echo OK 2023-12-25T20:55:09.736 INFO:tasks.workunit.client.0.smithi191.stdout:OK 2023-12-25T20:55:09.737 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-25T20:55:09.737 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-25T20:55:09.764 INFO:tasks.workunit:Running workunits matching mon/crush_ops.sh on client.0... 2023-12-25T20:55:09.765 INFO:tasks.workunit:Running workunit mon/crush_ops.sh... 2023-12-25T20:55:09.765 DEBUG:teuthology.orchestra.run.smithi191:workunit test mon/crush_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh 2023-12-25T20:55:09.827 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd crush dump 2023-12-25T20:55:10.829 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2023-12-25T20:55:10.829 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": [ 2023-12-25T20:55:10.829 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.829 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2023-12-25T20:55:10.829 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.0", 2023-12-25T20:55:10.829 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "ssd" 2023-12-25T20:55:10.830 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.830 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.830 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2023-12-25T20:55:10.830 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.1", 2023-12-25T20:55:10.830 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "ssd" 2023-12-25T20:55:10.830 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.830 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.830 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2023-12-25T20:55:10.830 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.2", 2023-12-25T20:55:10.830 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "ssd" 2023-12-25T20:55:10.830 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.830 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.830 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 3, 2023-12-25T20:55:10.830 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.3", 2023-12-25T20:55:10.831 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "ssd" 2023-12-25T20:55:10.831 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.831 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.831 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 4, 2023-12-25T20:55:10.831 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.4", 2023-12-25T20:55:10.831 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "ssd" 2023-12-25T20:55:10.831 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.831 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.831 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 5, 2023-12-25T20:55:10.831 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.5", 2023-12-25T20:55:10.831 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "ssd" 2023-12-25T20:55:10.831 INFO:tasks.workunit.client.0.smithi191.stdout: } 2023-12-25T20:55:10.831 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2023-12-25T20:55:10.831 INFO:tasks.workunit.client.0.smithi191.stdout: "types": [ 2023-12-25T20:55:10.831 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.832 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 0, 2023-12-25T20:55:10.832 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd" 2023-12-25T20:55:10.832 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.832 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.832 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2023-12-25T20:55:10.832 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "host" 2023-12-25T20:55:10.832 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.832 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.832 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 2, 2023-12-25T20:55:10.832 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "chassis" 2023-12-25T20:55:10.832 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.832 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.832 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 3, 2023-12-25T20:55:10.832 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "rack" 2023-12-25T20:55:10.832 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.833 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.833 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 4, 2023-12-25T20:55:10.833 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "row" 2023-12-25T20:55:10.833 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.833 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.833 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 5, 2023-12-25T20:55:10.833 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "pdu" 2023-12-25T20:55:10.833 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.833 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.833 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 6, 2023-12-25T20:55:10.833 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "pod" 2023-12-25T20:55:10.833 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.833 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.833 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 7, 2023-12-25T20:55:10.834 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "room" 2023-12-25T20:55:10.834 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.834 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.834 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 8, 2023-12-25T20:55:10.834 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "datacenter" 2023-12-25T20:55:10.834 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.834 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.834 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 9, 2023-12-25T20:55:10.834 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "zone" 2023-12-25T20:55:10.834 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.834 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.834 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 10, 2023-12-25T20:55:10.834 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "region" 2023-12-25T20:55:10.834 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.834 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.835 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2023-12-25T20:55:10.835 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "root" 2023-12-25T20:55:10.835 INFO:tasks.workunit.client.0.smithi191.stdout: } 2023-12-25T20:55:10.835 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2023-12-25T20:55:10.835 INFO:tasks.workunit.client.0.smithi191.stdout: "buckets": [ 2023-12-25T20:55:10.835 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.835 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -1, 2023-12-25T20:55:10.835 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "default", 2023-12-25T20:55:10.835 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2023-12-25T20:55:10.835 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "root", 2023-12-25T20:55:10.835 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 34560, 2023-12-25T20:55:10.835 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2023-12-25T20:55:10.835 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2023-12-25T20:55:10.835 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2023-12-25T20:55:10.836 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.836 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -3, 2023-12-25T20:55:10.836 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 17280, 2023-12-25T20:55:10.836 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2023-12-25T20:55:10.836 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.836 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.836 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -5, 2023-12-25T20:55:10.836 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 17280, 2023-12-25T20:55:10.836 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 1 2023-12-25T20:55:10.836 INFO:tasks.workunit.client.0.smithi191.stdout: } 2023-12-25T20:55:10.836 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2023-12-25T20:55:10.836 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.836 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.836 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -2, 2023-12-25T20:55:10.836 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "default~ssd", 2023-12-25T20:55:10.837 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2023-12-25T20:55:10.837 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "root", 2023-12-25T20:55:10.837 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 34560, 2023-12-25T20:55:10.837 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2023-12-25T20:55:10.837 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2023-12-25T20:55:10.837 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2023-12-25T20:55:10.837 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.837 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -4, 2023-12-25T20:55:10.837 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 17280, 2023-12-25T20:55:10.837 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2023-12-25T20:55:10.837 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.837 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.837 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -6, 2023-12-25T20:55:10.837 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 17280, 2023-12-25T20:55:10.838 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 1 2023-12-25T20:55:10.838 INFO:tasks.workunit.client.0.smithi191.stdout: } 2023-12-25T20:55:10.838 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2023-12-25T20:55:10.838 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.838 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.838 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -3, 2023-12-25T20:55:10.838 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "smithi191", 2023-12-25T20:55:10.838 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2023-12-25T20:55:10.838 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "host", 2023-12-25T20:55:10.838 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 17280, 2023-12-25T20:55:10.838 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2023-12-25T20:55:10.838 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2023-12-25T20:55:10.838 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2023-12-25T20:55:10.838 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.838 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 4, 2023-12-25T20:55:10.838 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 5760, 2023-12-25T20:55:10.839 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2023-12-25T20:55:10.839 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.839 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.839 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 3, 2023-12-25T20:55:10.839 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 5760, 2023-12-25T20:55:10.839 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 1 2023-12-25T20:55:10.839 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.839 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.839 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 5, 2023-12-25T20:55:10.839 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 5760, 2023-12-25T20:55:10.839 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 2 2023-12-25T20:55:10.839 INFO:tasks.workunit.client.0.smithi191.stdout: } 2023-12-25T20:55:10.839 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2023-12-25T20:55:10.839 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.839 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.840 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -4, 2023-12-25T20:55:10.840 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "smithi191~ssd", 2023-12-25T20:55:10.840 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2023-12-25T20:55:10.840 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "host", 2023-12-25T20:55:10.840 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 17280, 2023-12-25T20:55:10.840 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2023-12-25T20:55:10.840 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2023-12-25T20:55:10.840 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2023-12-25T20:55:10.840 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.840 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 4, 2023-12-25T20:55:10.840 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 5760, 2023-12-25T20:55:10.840 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2023-12-25T20:55:10.840 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.840 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.840 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 3, 2023-12-25T20:55:10.841 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 5760, 2023-12-25T20:55:10.841 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 1 2023-12-25T20:55:10.841 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.841 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.841 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 5, 2023-12-25T20:55:10.841 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 5760, 2023-12-25T20:55:10.841 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 2 2023-12-25T20:55:10.841 INFO:tasks.workunit.client.0.smithi191.stdout: } 2023-12-25T20:55:10.841 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2023-12-25T20:55:10.841 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.841 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.841 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -5, 2023-12-25T20:55:10.841 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "smithi153", 2023-12-25T20:55:10.841 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2023-12-25T20:55:10.841 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "host", 2023-12-25T20:55:10.842 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 17280, 2023-12-25T20:55:10.842 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2023-12-25T20:55:10.842 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2023-12-25T20:55:10.842 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2023-12-25T20:55:10.842 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.842 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2023-12-25T20:55:10.842 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 5760, 2023-12-25T20:55:10.842 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2023-12-25T20:55:10.842 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.842 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.842 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2023-12-25T20:55:10.842 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 5760, 2023-12-25T20:55:10.842 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 1 2023-12-25T20:55:10.842 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.843 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.843 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2023-12-25T20:55:10.843 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 5760, 2023-12-25T20:55:10.843 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 2 2023-12-25T20:55:10.843 INFO:tasks.workunit.client.0.smithi191.stdout: } 2023-12-25T20:55:10.843 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2023-12-25T20:55:10.843 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.843 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.843 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -6, 2023-12-25T20:55:10.843 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "smithi153~ssd", 2023-12-25T20:55:10.843 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2023-12-25T20:55:10.843 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "host", 2023-12-25T20:55:10.843 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 17280, 2023-12-25T20:55:10.843 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2023-12-25T20:55:10.843 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2023-12-25T20:55:10.844 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2023-12-25T20:55:10.844 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.844 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2023-12-25T20:55:10.844 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 5760, 2023-12-25T20:55:10.844 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2023-12-25T20:55:10.844 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.844 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.844 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2023-12-25T20:55:10.844 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 5760, 2023-12-25T20:55:10.844 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 1 2023-12-25T20:55:10.844 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.844 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.844 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2023-12-25T20:55:10.844 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 5760, 2023-12-25T20:55:10.844 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 2 2023-12-25T20:55:10.844 INFO:tasks.workunit.client.0.smithi191.stdout: } 2023-12-25T20:55:10.845 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2023-12-25T20:55:10.845 INFO:tasks.workunit.client.0.smithi191.stdout: } 2023-12-25T20:55:10.845 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2023-12-25T20:55:10.845 INFO:tasks.workunit.client.0.smithi191.stdout: "rules": [ 2023-12-25T20:55:10.845 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.845 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_id": 0, 2023-12-25T20:55:10.845 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_name": "replicated_rule", 2023-12-25T20:55:10.846 INFO:tasks.workunit.client.0.smithi191.stdout: "ruleset": 0, 2023-12-25T20:55:10.847 INFO:tasks.workunit.client.0.smithi191.stdout: "type": 1, 2023-12-25T20:55:10.847 INFO:tasks.workunit.client.0.smithi191.stdout: "min_size": 1, 2023-12-25T20:55:10.847 INFO:tasks.workunit.client.0.smithi191.stdout: "max_size": 10, 2023-12-25T20:55:10.847 INFO:tasks.workunit.client.0.smithi191.stdout: "steps": [ 2023-12-25T20:55:10.847 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.847 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "take", 2023-12-25T20:55:10.847 INFO:tasks.workunit.client.0.smithi191.stdout: "item": -1, 2023-12-25T20:55:10.847 INFO:tasks.workunit.client.0.smithi191.stdout: "item_name": "default" 2023-12-25T20:55:10.847 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.847 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.847 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "choose_firstn", 2023-12-25T20:55:10.847 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 0, 2023-12-25T20:55:10.847 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "osd" 2023-12-25T20:55:10.848 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.848 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.848 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "emit" 2023-12-25T20:55:10.848 INFO:tasks.workunit.client.0.smithi191.stdout: } 2023-12-25T20:55:10.848 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2023-12-25T20:55:10.848 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.848 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.848 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_id": 1, 2023-12-25T20:55:10.848 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_name": "erasure-code", 2023-12-25T20:55:10.848 INFO:tasks.workunit.client.0.smithi191.stdout: "ruleset": 1, 2023-12-25T20:55:10.848 INFO:tasks.workunit.client.0.smithi191.stdout: "type": 3, 2023-12-25T20:55:10.848 INFO:tasks.workunit.client.0.smithi191.stdout: "min_size": 3, 2023-12-25T20:55:10.848 INFO:tasks.workunit.client.0.smithi191.stdout: "max_size": 3, 2023-12-25T20:55:10.848 INFO:tasks.workunit.client.0.smithi191.stdout: "steps": [ 2023-12-25T20:55:10.848 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.849 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "set_chooseleaf_tries", 2023-12-25T20:55:10.849 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 5 2023-12-25T20:55:10.849 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.849 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.849 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "set_choose_tries", 2023-12-25T20:55:10.849 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 100 2023-12-25T20:55:10.850 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.850 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.850 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "take", 2023-12-25T20:55:10.850 INFO:tasks.workunit.client.0.smithi191.stdout: "item": -1, 2023-12-25T20:55:10.851 INFO:tasks.workunit.client.0.smithi191.stdout: "item_name": "default" 2023-12-25T20:55:10.851 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.851 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.851 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "choose_indep", 2023-12-25T20:55:10.851 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 0, 2023-12-25T20:55:10.851 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "osd" 2023-12-25T20:55:10.851 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.852 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:10.852 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "emit" 2023-12-25T20:55:10.852 INFO:tasks.workunit.client.0.smithi191.stdout: } 2023-12-25T20:55:10.852 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2023-12-25T20:55:10.852 INFO:tasks.workunit.client.0.smithi191.stdout: } 2023-12-25T20:55:10.852 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2023-12-25T20:55:10.852 INFO:tasks.workunit.client.0.smithi191.stdout: "tunables": { 2023-12-25T20:55:10.852 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_local_tries": 0, 2023-12-25T20:55:10.852 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_local_fallback_tries": 0, 2023-12-25T20:55:10.852 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_total_tries": 50, 2023-12-25T20:55:10.852 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_descend_once": 1, 2023-12-25T20:55:10.852 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_vary_r": 1, 2023-12-25T20:55:10.852 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_stable": 1, 2023-12-25T20:55:10.853 INFO:tasks.workunit.client.0.smithi191.stdout: "straw_calc_version": 1, 2023-12-25T20:55:10.853 INFO:tasks.workunit.client.0.smithi191.stdout: "allowed_bucket_algs": 54, 2023-12-25T20:55:10.853 INFO:tasks.workunit.client.0.smithi191.stdout: "profile": "jewel", 2023-12-25T20:55:10.853 INFO:tasks.workunit.client.0.smithi191.stdout: "optimal_tunables": 1, 2023-12-25T20:55:10.853 INFO:tasks.workunit.client.0.smithi191.stdout: "legacy_tunables": 0, 2023-12-25T20:55:10.853 INFO:tasks.workunit.client.0.smithi191.stdout: "minimum_required_version": "jewel", 2023-12-25T20:55:10.853 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables": 1, 2023-12-25T20:55:10.853 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables2": 1, 2023-12-25T20:55:10.853 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v2_rules": 1, 2023-12-25T20:55:10.853 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables3": 1, 2023-12-25T20:55:10.853 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v3_rules": 0, 2023-12-25T20:55:10.853 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v4_buckets": 1, 2023-12-25T20:55:10.853 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables5": 1, 2023-12-25T20:55:10.853 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v5_rules": 0 2023-12-25T20:55:10.853 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:10.853 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_args": {} 2023-12-25T20:55:10.854 INFO:tasks.workunit.client.0.smithi191.stdout:} 2023-12-25T20:55:10.854 INFO:tasks.workunit.client.0.smithi191.stdout: 2023-12-25T20:55:10.854 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd crush rule dump 2023-12-25T20:55:11.219 INFO:tasks.workunit.client.0.smithi191.stdout:[ 2023-12-25T20:55:11.219 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:11.219 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_id": 0, 2023-12-25T20:55:11.219 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_name": "replicated_rule", 2023-12-25T20:55:11.219 INFO:tasks.workunit.client.0.smithi191.stdout: "ruleset": 0, 2023-12-25T20:55:11.219 INFO:tasks.workunit.client.0.smithi191.stdout: "type": 1, 2023-12-25T20:55:11.219 INFO:tasks.workunit.client.0.smithi191.stdout: "min_size": 1, 2023-12-25T20:55:11.219 INFO:tasks.workunit.client.0.smithi191.stdout: "max_size": 10, 2023-12-25T20:55:11.219 INFO:tasks.workunit.client.0.smithi191.stdout: "steps": [ 2023-12-25T20:55:11.219 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:11.220 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "take", 2023-12-25T20:55:11.220 INFO:tasks.workunit.client.0.smithi191.stdout: "item": -1, 2023-12-25T20:55:11.220 INFO:tasks.workunit.client.0.smithi191.stdout: "item_name": "default" 2023-12-25T20:55:11.220 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:11.220 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:11.220 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "choose_firstn", 2023-12-25T20:55:11.220 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 0, 2023-12-25T20:55:11.220 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "osd" 2023-12-25T20:55:11.220 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:11.220 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:11.220 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "emit" 2023-12-25T20:55:11.221 INFO:tasks.workunit.client.0.smithi191.stdout: } 2023-12-25T20:55:11.221 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2023-12-25T20:55:11.221 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:11.221 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:11.221 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_id": 1, 2023-12-25T20:55:11.221 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_name": "erasure-code", 2023-12-25T20:55:11.221 INFO:tasks.workunit.client.0.smithi191.stdout: "ruleset": 1, 2023-12-25T20:55:11.221 INFO:tasks.workunit.client.0.smithi191.stdout: "type": 3, 2023-12-25T20:55:11.221 INFO:tasks.workunit.client.0.smithi191.stdout: "min_size": 3, 2023-12-25T20:55:11.221 INFO:tasks.workunit.client.0.smithi191.stdout: "max_size": 3, 2023-12-25T20:55:11.221 INFO:tasks.workunit.client.0.smithi191.stdout: "steps": [ 2023-12-25T20:55:11.221 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:11.221 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "set_chooseleaf_tries", 2023-12-25T20:55:11.221 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 5 2023-12-25T20:55:11.222 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:11.222 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:11.222 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "set_choose_tries", 2023-12-25T20:55:11.222 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 100 2023-12-25T20:55:11.222 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:11.222 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:11.222 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "take", 2023-12-25T20:55:11.222 INFO:tasks.workunit.client.0.smithi191.stdout: "item": -1, 2023-12-25T20:55:11.222 INFO:tasks.workunit.client.0.smithi191.stdout: "item_name": "default" 2023-12-25T20:55:11.222 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:11.222 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:11.223 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "choose_indep", 2023-12-25T20:55:11.223 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 0, 2023-12-25T20:55:11.223 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "osd" 2023-12-25T20:55:11.223 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2023-12-25T20:55:11.223 INFO:tasks.workunit.client.0.smithi191.stdout: { 2023-12-25T20:55:11.223 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "emit" 2023-12-25T20:55:11.223 INFO:tasks.workunit.client.0.smithi191.stdout: } 2023-12-25T20:55:11.223 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2023-12-25T20:55:11.223 INFO:tasks.workunit.client.0.smithi191.stdout: } 2023-12-25T20:55:11.223 INFO:tasks.workunit.client.0.smithi191.stdout:] 2023-12-25T20:55:11.223 INFO:tasks.workunit.client.0.smithi191.stdout: 2023-12-25T20:55:11.230 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd crush rule ls 2023-12-25T20:55:12.450 INFO:tasks.workunit.client.0.smithi191.stdout:replicated_rule 2023-12-25T20:55:12.451 INFO:tasks.workunit.client.0.smithi191.stdout:erasure-code 2023-12-25T20:55:12.463 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd crush rule list 2023-12-25T20:55:12.842 INFO:tasks.workunit.client.0.smithi191.stdout:replicated_rule 2023-12-25T20:55:12.843 INFO:tasks.workunit.client.0.smithi191.stdout:erasure-code 2023-12-25T20:55:12.854 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd crush rule create-simple foo default host 2023-12-25T20:55:13.816 INFO:tasks.workunit.client.0.smithi191.stderr:rule foo already exists 2023-12-25T20:55:13.826 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd crush rule create-simple foo default host 2023-12-25T20:55:14.209 INFO:tasks.workunit.client.0.smithi191.stderr:rule foo already exists 2023-12-25T20:55:14.218 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd crush rule create-simple bar default host 2023-12-25T20:55:14.821 INFO:tasks.workunit.client.0.smithi191.stderr:rule bar already exists 2023-12-25T20:55:14.831 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd crush rm-device-class all 2023-12-25T20:55:15.433 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2023-12-25T20:55:15.434 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2023-12-25T20:55:15.434 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-25T20:55:15.434 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-12-25T20:55:15.437 INFO:tasks.ceph.mon.c:Started 2023-12-25T20:55:15.437 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-25T20:55:15.437 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:55:15.810 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":34,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:55:16.378 INFO:tasks.workunit.client.0.smithi191.stderr:osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): 2023-12-25T20:55:16.389 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd crush set-device-class ssd osd.0 2023-12-25T20:55:18.510 INFO:tasks.workunit.client.0.smithi191.stderr:osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' 2023-12-25T20:55:18.522 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd crush set-device-class hdd osd.1 2023-12-25T20:55:18.824 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:55:19.190 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":34,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:55:20.520 INFO:tasks.workunit.client.0.smithi191.stderr:osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' 2023-12-25T20:55:20.532 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd crush rule create-replicated foo-ssd default host ssd 2023-12-25T20:55:22.204 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:55:25.909 INFO:tasks.workunit.client.0.smithi191.stderr:rule foo-ssd already exists 2023-12-25T20:55:25.918 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd crush rule create-replicated foo-hdd default host hdd 2023-12-25T20:55:25.984 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":38,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:55:27.042 INFO:tasks.workunit.client.0.smithi191.stderr:rule foo-hdd already exists 2023-12-25T20:55:27.052 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd crush rule ls-by-class ssd 2023-12-25T20:55:27.052 INFO:tasks.workunit.client.0.smithi191.stderr:+ grep foo-ssd 2023-12-25T20:55:27.443 INFO:tasks.workunit.client.0.smithi191.stdout:foo-ssd 2023-12-25T20:55:27.444 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd crush rule ls-by-class ssd 2023-12-25T20:55:27.444 INFO:tasks.workunit.client.0.smithi191.stderr:+ expect_false grep foo-hdd 2023-12-25T20:55:27.444 INFO:tasks.workunit.client.0.smithi191.stderr:+ set -x 2023-12-25T20:55:27.444 INFO:tasks.workunit.client.0.smithi191.stderr:+ grep foo-hdd 2023-12-25T20:55:27.844 INFO:tasks.workunit.client.0.smithi191.stderr:+ return 0 2023-12-25T20:55:27.844 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd crush rule ls-by-class hdd 2023-12-25T20:55:27.844 INFO:tasks.workunit.client.0.smithi191.stderr:+ grep foo-hdd 2023-12-25T20:55:28.235 INFO:tasks.workunit.client.0.smithi191.stdout:foo-hdd 2023-12-25T20:55:28.236 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd crush rule ls-by-class hdd 2023-12-25T20:55:28.236 INFO:tasks.workunit.client.0.smithi191.stderr:+ expect_false grep foo-ssd 2023-12-25T20:55:28.236 INFO:tasks.workunit.client.0.smithi191.stderr:+ set -x 2023-12-25T20:55:28.236 INFO:tasks.workunit.client.0.smithi191.stderr:+ grep foo-ssd 2023-12-25T20:55:28.629 INFO:tasks.workunit.client.0.smithi191.stderr:+ return 0 2023-12-25T20:55:28.629 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd erasure-code-profile set ec-foo-ssd crush-device-class=ssd m=2 k=2 2023-12-25T20:55:28.998 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:55:29.188 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool create ec-foo 2 erasure ec-foo-ssd 2023-12-25T20:55:29.368 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:55:29.381 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-25T20:55:29.381 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-25T20:55:29.381 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-25T20:55:29.518 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:55:29.518 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:55:29.518 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:55:29.518 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2023-12-25T20:55:29.518 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 40, 2023-12-25T20:55:29.518 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:55:29.518 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:55:29.518 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:55:29.518 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:55:29.518 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:29.518 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2023-12-25T20:55:29.518 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:55:29.519 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:55:29.519 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:55:29.519 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:29.519 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:29.519 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:29.519 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:29.519 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:29.519 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:29.519 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:29.519 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:29.519 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:29.519 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:55:29.519 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:55:29.519 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:29.520 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:29.520 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:29.520 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:29.520 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:29.520 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:29.520 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:29.520 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:29.520 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:29.520 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:29.520 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:55:29.520 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:55:29.520 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:55:29.520 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:55:29.520 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:55:29.520 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:55:29.521 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:55:29.521 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:55:29.521 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:55:29.521 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:55:29.521 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:55:29.521 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:55:29.521 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:55:29.521 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:55:29.521 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:55:29.521 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:55:29.521 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:55:29.521 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:29.522 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:29.522 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:29.522 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:29.522 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:29.522 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:29.522 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:29.522 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:29.522 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:29.522 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:55:29.522 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:29.522 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:55:29.522 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:29.522 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:55:29.523 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:55:29.523 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:29.523 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:29.523 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:29.523 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:29.523 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:55:29.523 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:29.523 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:29.523 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:29.523 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:29.523 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:55:29.523 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:29.523 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:29.523 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:29.524 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:29.524 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:55:29.524 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:55:29.524 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:29.524 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:29.524 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:29.524 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:29.524 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:29.524 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:55:29.524 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:55:29.524 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:29.524 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:29.524 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:29.524 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:29.525 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:55:29.525 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:29.525 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:29.525 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:29.525 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:29.525 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:55:29.525 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:29.525 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:29.525 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:29.525 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:29.525 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:55:29.525 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:55:29.525 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:29.525 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:29.526 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:29.526 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:29.526 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:29.526 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:55:29.526 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:55:29.526 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:29.526 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:29.526 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:29.526 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:29.526 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:55:29.526 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:29.526 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:29.526 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:29.526 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:29.526 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:55:29.527 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:29.527 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:29.527 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:29.527 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:29.527 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:55:29.527 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:55:29.527 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:29.527 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:29.527 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:29.527 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:29.527 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:29.527 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:29.527 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:55:29.527 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:55:29.527 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:29.528 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:29.528 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:29.528 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:55:29.528 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:29.528 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:29.528 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:55:29.528 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:29.528 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:29.528 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:29.528 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2023-12-25T20:55:29.528 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:29.528 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:29.529 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:55:29.529 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:29.529 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:29.529 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:29.529 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2023-12-25T20:55:29.529 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:29.529 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:29.529 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:29.529 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:55:29.529 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:55:29.530 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:55:29.530 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-25T20:55:29.531 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-25T20:55:31.073 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:55:31.073 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:55:31.074 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:55:31.074 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2023-12-25T20:55:31.074 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 40, 2023-12-25T20:55:31.074 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:55:31.074 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:55:31.074 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:55:31.074 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:55:31.074 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:31.074 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 4, 2023-12-25T20:55:31.074 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:55:31.074 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:55:31.074 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:55:31.074 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:31.074 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:31.075 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:31.075 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:31.075 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:31.075 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:31.075 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:31.075 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:31.075 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:31.075 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:55:31.075 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:55:31.075 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:31.075 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:31.075 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:31.075 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:31.075 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:31.076 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:31.076 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:31.076 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:31.076 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:31.076 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.076 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:55:31.076 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:55:31.076 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:55:31.076 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:55:31.076 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:55:31.076 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:55:31.076 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:55:31.076 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:55:31.077 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:55:31.077 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:55:31.077 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:55:31.077 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:55:31.077 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:55:31.077 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:55:31.077 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:55:31.077 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:55:31.077 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:55:31.077 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:31.077 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:31.077 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:31.077 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:31.078 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:31.078 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:31.078 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:31.078 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:31.078 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:31.078 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:55:31.078 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.078 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:55:31.078 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.078 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:55:31.078 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:55:31.078 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:31.078 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:31.079 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.079 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:31.079 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:55:31.079 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:31.079 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.079 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.079 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:31.079 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:55:31.079 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:31.079 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:31.079 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:31.079 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.079 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:55:31.079 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:55:31.079 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:31.080 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:31.080 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:31.080 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.080 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.080 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:55:31.080 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:55:31.080 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:31.080 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:31.080 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.080 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:31.080 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:55:31.080 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:31.080 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.081 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.081 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:31.081 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:55:31.081 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:31.081 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:31.081 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:31.081 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.081 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:55:31.081 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:55:31.081 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:31.081 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:31.081 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:31.081 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.081 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.081 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:55:31.082 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:55:31.082 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:31.082 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:31.082 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.082 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:31.082 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:55:31.082 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:31.082 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.082 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.082 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:31.082 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:55:31.082 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:31.082 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:31.082 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:31.082 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.083 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:55:31.083 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:55:31.083 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:31.083 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:31.083 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:31.083 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:31.083 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:31.083 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.083 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:55:31.083 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:55:31.083 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.083 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:31.083 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:31.083 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:55:31.083 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:31.084 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:31.084 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:55:31.084 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.084 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:31.084 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:31.084 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:55:31.084 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:31.084 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:31.084 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.084 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:55:31.084 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:55:31.088 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:55:31.088 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-25T20:55:31.088 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-25T20:55:31.182 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'ec-foo' already exists 2023-12-25T20:55:31.192 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd pool rm ec-foo ec-foo --yes-i-really-really-mean-it 2023-12-25T20:55:31.234 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:55:31.235 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:55:31.235 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:55:31.235 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2023-12-25T20:55:31.235 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 40, 2023-12-25T20:55:31.235 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:55:31.235 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:55:31.235 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:55:31.235 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:55:31.235 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:31.235 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 4, 2023-12-25T20:55:31.235 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:55:31.235 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:55:31.235 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:55:31.235 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:31.236 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:31.236 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:31.236 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:31.236 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:31.236 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:31.236 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:31.236 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:31.236 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:31.236 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:55:31.236 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:55:31.236 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:31.237 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:31.237 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:31.237 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:31.237 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:31.237 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:31.237 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:31.237 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:31.237 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:31.237 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.237 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:55:31.237 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:55:31.238 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:55:31.238 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:55:31.238 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:55:31.238 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:55:31.238 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:55:31.238 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:55:31.238 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:55:31.238 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:55:31.238 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:55:31.238 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:55:31.238 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:55:31.238 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:55:31.238 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:55:31.238 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:55:31.239 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:55:31.239 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:31.239 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:31.239 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:31.239 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:31.239 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:31.239 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:31.239 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:31.239 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:31.239 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:31.239 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:55:31.239 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.240 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:55:31.240 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.240 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:55:31.240 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:55:31.240 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:31.240 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:31.240 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.240 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:31.240 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:55:31.240 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:31.240 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.240 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.240 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:31.240 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:55:31.240 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:31.241 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:31.241 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:31.241 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.241 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:55:31.241 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:55:31.241 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:31.241 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:31.241 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:31.241 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.241 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.241 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:55:31.241 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:55:31.241 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:31.242 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:31.242 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.242 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:31.242 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:55:31.242 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:31.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.242 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.242 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:31.242 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:55:31.242 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:31.242 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:31.242 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:31.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.242 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:55:31.242 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:55:31.243 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:31.243 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:31.243 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:31.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.243 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.243 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:55:31.243 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:55:31.243 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:31.243 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:31.243 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.243 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:31.243 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:55:31.243 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:31.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.243 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.243 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:31.244 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:55:31.244 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:31.244 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:31.244 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:31.244 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.244 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:55:31.244 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:55:31.244 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:31.244 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:31.244 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:31.244 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:31.244 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:31.244 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.244 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:55:31.244 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:55:31.245 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.245 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:31.245 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:31.245 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:55:31.245 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:31.245 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:31.245 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:55:31.245 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.245 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:31.245 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:31.245 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2023-12-25T20:55:31.245 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:31.245 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:31.246 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:55:31.246 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.246 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:31.246 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:31.246 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2023-12-25T20:55:31.246 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:31.246 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:31.246 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2023-12-25T20:55:31.246 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:31.246 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:31.246 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:31.246 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:55:31.246 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:31.246 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:31.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:31.247 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:55:31.247 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:55:31.248 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:55:31.248 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-12-25T20:55:31.248 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-12-25T20:55:31.609 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:55:31.608+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:55:31.609 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:55:31.608+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=10,kv=5,logm=71} crc {auth=254922924,config=3140514010,health=4291592628,kv=691660919,logm=1556365890}) 2023-12-25T20:55:31.609 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:55:31.608+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=12,config=2,health=11,kv=6,logm=69} crc {auth=254922924,config=3140514009,health=851079395,kv=661943861,logm=3588624524}) 2023-12-25T20:55:31.610 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:55:31.608+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:55:31.610 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:55:31.608+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=10,kv=5,logm=71} crc {auth=254922924,config=3140514010,health=4291592628,kv=691660919,logm=1556365890}) 2023-12-25T20:55:31.610 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:55:31.608+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=11,config=2,health=10,kv=6,logm=71} crc {auth=3789660155,config=3140514009,health=882794937,kv=661943861,logm=3020018008}) 2023-12-25T20:55:31.612 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:55:31.611+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:55:31.612 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:55:31.611+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2190885473}) 2023-12-25T20:55:31.612 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:55:31.611+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=743836636}) 2023-12-25T20:55:31.612 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:55:31.611+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:55:31.612 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:55:31.611+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2190885473}) 2023-12-25T20:55:31.612 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:55:31.611+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1615446648}) 2023-12-25T20:55:31.613 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:55:31.612+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:55:31.613 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:55:31.612+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=6,mdsmap=3,mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=2,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=63} crc {logm=289337500,mdsmap=1531151339,mgr=3403703901,mgr_command_descs=1448462394,mgr_metadata=1482708640,mgrstat=3835014475,mon_config_key=1750829095,monmap=3474379420,osd_metadata=1683759159,osd_pg_creating=3959274828,osdmap=2286739647}) 2023-12-25T20:55:31.613 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:55:31.612+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=6,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=9,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=61} crc {logm=289337500,mdsmap=1531151339,mgr=2949565714,mgr_metadata=1482708640,mgrstat=303480603,mon_config_key=2934451455,monmap=3296645973,osd_metadata=1683759159,osd_pg_creating=3959274828,osdmap=1749328601}) 2023-12-25T20:55:31.613 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:55:31.612+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:55:31.613 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:55:31.612+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=6,mdsmap=3,mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=2,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=63} crc {logm=289337500,mdsmap=1531151339,mgr=3403703901,mgr_command_descs=1448462394,mgr_metadata=1482708640,mgrstat=3835014475,mon_config_key=1750829095,monmap=3474379420,osd_metadata=1683759159,osd_pg_creating=3959274828,osdmap=2286739647}) 2023-12-25T20:55:31.613 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:55:31.612+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=6,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=2,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=60} crc {logm=289337500,mdsmap=1531151339,mgr=2949565714,mgr_command_descs=1448462394,mgr_metadata=1482708640,mgrstat=3835014475,mon_config_key=1750829095,monmap=3296645973,osd_metadata=4118341297,osd_pg_creating=3959274828,osdmap=2257272072}) 2023-12-25T20:55:31.616 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:55:31.615+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:55:31.616 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:55:31.616+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=44} crc {osdmap=3246128990}) 2023-12-25T20:55:31.616 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:55:31.616+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=42} crc {osdmap=600380821}) 2023-12-25T20:55:31.626 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-12-25T20:55:32.501 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'ec-foo' does not exist 2023-12-25T20:55:32.511 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd crush rule ls 2023-12-25T20:55:32.511 INFO:tasks.workunit.client.0.smithi191.stderr:+ grep foo 2023-12-25T20:55:32.626 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-25T20:55:32.627 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:55:32.902 INFO:tasks.workunit.client.0.smithi191.stdout:foo 2023-12-25T20:55:32.902 INFO:tasks.workunit.client.0.smithi191.stdout:foo-ssd 2023-12-25T20:55:32.902 INFO:tasks.workunit.client.0.smithi191.stdout:foo-hdd 2023-12-25T20:55:32.902 INFO:tasks.workunit.client.0.smithi191.stdout:ec-foo 2023-12-25T20:55:32.902 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd crush rule rename foo foo-asdf 2023-12-25T20:55:32.993 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:55:33.006 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-25T20:55:33.007 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-12-25T20:55:33.007 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-25T20:55:33.007 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-25T20:55:33.144 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:55:33.144 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:55:33.144 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:55:33.144 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2023-12-25T20:55:33.144 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 40, 2023-12-25T20:55:33.144 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:55:33.144 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:55:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:55:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:55:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 6, 2023-12-25T20:55:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:55:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:55:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:55:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:55:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:55:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:55:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:55:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:55:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:55:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:55:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:55:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:55:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:55:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:55:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:55:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:55:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:55:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:55:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:55:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:55:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:55:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:55:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:55:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:55:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:55:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:55:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:55:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:55:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:55:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:55:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:55:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:55:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:55:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:55:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:55:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:55:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:55:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:55:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:55:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:55:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:55:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:55:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:55:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:55:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:55:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:55:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2023-12-25T20:55:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:55:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.155 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:33.155 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:33.155 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2023-12-25T20:55:33.155 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:33.155 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:33.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.155 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:55:33.155 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:55:33.156 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:55:33.156 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-25T20:55:33.156 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-25T20:55:33.293 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:55:33.293 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:55:33.294 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:55:33.294 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2023-12-25T20:55:33.294 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 40, 2023-12-25T20:55:33.294 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:55:33.294 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:55:33.294 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:55:33.294 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:55:33.294 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:33.294 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 6, 2023-12-25T20:55:33.294 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:55:33.294 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:55:33.294 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:55:33.294 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:33.294 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:33.295 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:33.295 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:33.295 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:33.295 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:33.295 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:33.295 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:33.295 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:33.295 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:55:33.295 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:55:33.295 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:33.295 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:33.295 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:33.295 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:33.295 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:33.295 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:33.296 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:33.296 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:33.296 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:33.296 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.296 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:55:33.296 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:55:33.296 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:55:33.296 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:55:33.296 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:55:33.296 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:55:33.296 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:55:33.296 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:55:33.296 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:55:33.296 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:55:33.296 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:55:33.297 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:55:33.297 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:55:33.297 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:55:33.297 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:55:33.297 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:55:33.297 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:55:33.297 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:33.297 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:33.297 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:33.297 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:33.297 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:33.297 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:33.297 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:33.297 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:33.297 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:33.298 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:55:33.298 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.298 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:55:33.298 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.298 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:55:33.298 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:55:33.298 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:33.298 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:33.298 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.298 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:33.298 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:55:33.298 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:33.298 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.298 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.298 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:55:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:55:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:55:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:55:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:55:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:33.300 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.300 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:33.300 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:55:33.300 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:33.300 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.300 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.300 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:33.300 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:55:33.300 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:33.300 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:33.300 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:33.300 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.300 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:55:33.300 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:55:33.300 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:33.301 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:33.301 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:33.301 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.301 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.301 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:55:33.301 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:55:33.301 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:33.301 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:33.301 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.301 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:33.301 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:55:33.301 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:33.301 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.301 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.301 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:55:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:55:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:55:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:55:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:55:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.303 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:33.303 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:33.303 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:55:33.303 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:33.303 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:33.303 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:55:33.303 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.303 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:33.303 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:33.303 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:55:33.303 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:33.303 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:33.303 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.303 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:55:33.303 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:55:33.305 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:55:33.305 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-25T20:55:33.305 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-25T20:55:33.441 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:55:33.441 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:55:33.441 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:55:33.442 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2023-12-25T20:55:33.442 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 40, 2023-12-25T20:55:33.442 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:55:33.442 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:55:33.442 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:55:33.442 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:55:33.442 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:33.442 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 6, 2023-12-25T20:55:33.442 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:55:33.442 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:55:33.442 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:55:33.442 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:33.442 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:33.442 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:33.442 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:33.443 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:33.443 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:33.443 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:33.443 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:33.443 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:33.443 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:55:33.443 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:55:33.443 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:33.443 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:33.443 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:33.443 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:33.443 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:33.443 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:33.443 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:33.444 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:33.444 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:33.444 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.444 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:55:33.444 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:55:33.444 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:55:33.444 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:55:33.444 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:55:33.444 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:55:33.444 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:55:33.444 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:55:33.444 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:55:33.444 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:55:33.444 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:55:33.444 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:55:33.445 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:55:33.445 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:55:33.445 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:55:33.445 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:55:33.445 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:55:33.445 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:33.445 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:33.445 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:33.445 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:33.445 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:33.445 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:33.445 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:33.445 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:33.445 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:33.445 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:55:33.446 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.446 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:55:33.446 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.446 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:55:33.446 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:55:33.446 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:33.446 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:33.446 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.446 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:33.446 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:55:33.446 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:33.446 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.446 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.446 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:33.447 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:55:33.447 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:33.447 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:33.447 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:33.447 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.447 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:55:33.447 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:55:33.447 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:33.447 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:33.447 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:33.447 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.447 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.447 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:55:33.447 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:55:33.447 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:33.448 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:33.448 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.448 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:33.448 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:55:33.448 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:33.448 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.448 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.448 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:33.448 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:55:33.448 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:33.448 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:33.448 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:33.448 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.448 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:55:33.448 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:55:33.448 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:33.449 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:33.449 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:33.449 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.449 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.449 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:55:33.449 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:55:33.449 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:33.449 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:33.449 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.449 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:33.449 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:55:33.449 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:33.449 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.449 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.449 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:55:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:55:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:55:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:55:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:55:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:55:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:55:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2023-12-25T20:55:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:55:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.452 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:33.452 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:33.452 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2023-12-25T20:55:33.452 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:33.452 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:33.452 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2023-12-25T20:55:33.452 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:33.452 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:33.452 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:33.452 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:55:33.452 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:33.452 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:33.452 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:33.452 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:55:33.452 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:55:33.454 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:55:33.454 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2023-12-25T20:55:33.454 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-12-25T20:55:33.454 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2023-12-25T20:55:33.454 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-12-25T20:55:33.454 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-25T20:55:33.454 INFO:teuthology.orchestra.run:waiting for 300 2023-12-25T20:55:33.466 INFO:tasks.ceph.mon.b:Stopped 2023-12-25T20:55:33.466 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-12-25T20:55:33.466 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:55:33.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-12-25T20:55:33.578 INFO:tasks.workunit.client.0.smithi191.stderr:already renamed to 'foo-asdf' 2023-12-25T20:55:33.587 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd crush rule rename foo foo-asdf 2023-12-25T20:55:33.824 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:55:33.976 INFO:tasks.workunit.client.0.smithi191.stderr:already renamed to 'foo-asdf' 2023-12-25T20:55:33.985 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd crush rule rename bar bar-asdf 2023-12-25T20:55:36.837 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:55:37.202 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:55:39.474 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-12-25T20:55:40.217 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:55:41.415 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:55:44.430 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:55:48.405 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":44,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:55:48.419 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-12-25T20:55:48.419 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-12-25T20:55:48.419 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-25T20:55:48.556 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:55:48.556 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:55:48.556 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:55:48.556 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2023-12-25T20:55:48.556 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 44, 2023-12-25T20:55:48.556 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:55:48.556 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:55:48.556 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:55:48.556 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:48.556 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2023-12-25T20:55:48.557 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:55:48.557 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:55:48.557 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:55:48.557 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:48.557 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:48.557 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:48.557 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:48.557 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:48.557 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:48.557 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:48.557 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:48.557 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:48.558 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:55:48.558 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:55:48.558 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:48.558 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:48.558 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:48.558 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:48.558 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:48.558 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:48.558 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:48.558 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:48.558 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:48.558 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.558 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:55:48.558 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:55:48.558 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:55:48.559 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:55:48.559 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:55:48.559 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:55:48.559 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:55:48.559 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:55:48.559 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:55:48.559 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:55:48.559 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:55:48.559 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:55:48.559 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:55:48.559 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:55:48.559 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:55:48.559 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:55:48.559 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:55:48.560 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:48.560 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:48.560 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:48.560 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:48.560 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:48.560 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:48.560 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:48.560 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:48.560 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:48.560 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:55:48.560 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.560 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:55:48.560 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.560 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:55:48.561 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:55:48.561 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:48.561 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:48.561 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.561 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:48.561 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:55:48.561 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:48.561 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.561 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.561 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:48.561 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:55:48.561 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:48.561 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:48.561 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:48.561 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:55:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:55:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:55:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:55:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:55:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:55:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:55:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:55:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:55:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:55:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:55:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:55:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:55:48.566 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:55:48.566 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:48.566 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:48.566 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:48.566 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:48.566 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:48.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.566 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:55:48.566 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:55:48.566 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.566 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:48.566 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:48.567 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:55:48.567 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:48.567 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:48.567 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:55:48.567 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.567 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:48.567 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:48.567 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2023-12-25T20:55:48.567 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:48.567 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:48.567 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:55:48.567 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.567 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:48.567 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:48.567 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2023-12-25T20:55:48.568 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:48.568 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:48.568 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.568 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:55:48.568 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:55:48.569 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-25T20:55:48.569 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-12-25T20:55:48.569 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-25T20:55:48.707 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:55:48.707 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:55:48.707 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:55:48.707 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2023-12-25T20:55:48.707 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 44, 2023-12-25T20:55:48.707 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:55:48.707 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:55:48.707 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:55:48.707 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:48.707 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2023-12-25T20:55:48.707 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:55:48.707 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:55:48.707 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:55:48.708 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:48.708 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:48.708 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:48.708 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:48.708 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:48.708 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:48.708 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:48.708 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:48.708 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:48.708 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:55:48.708 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:55:48.708 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:48.708 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:48.708 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:48.708 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:48.709 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:48.709 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:48.709 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:48.709 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:48.709 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:48.709 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.709 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:55:48.709 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:55:48.709 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:55:48.709 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:55:48.709 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:55:48.709 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:55:48.709 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:55:48.709 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:55:48.709 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:55:48.710 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:55:48.710 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:55:48.710 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:55:48.710 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:55:48.710 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:55:48.710 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:55:48.710 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:55:48.710 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:55:48.710 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:55:48.710 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:55:48.710 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:55:48.710 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:55:48.710 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:55:48.710 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:55:48.710 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:55:48.711 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:55:48.711 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:48.711 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:55:48.711 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.711 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:55:48.711 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.711 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:55:48.711 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:55:48.711 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:48.711 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:48.711 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.711 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:48.711 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:55:48.711 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:48.711 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.711 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.712 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:48.712 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:55:48.712 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:48.712 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:48.712 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:48.712 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.712 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:55:48.712 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:55:48.712 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:48.712 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:48.712 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:48.712 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.712 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.712 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:55:48.712 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:55:48.713 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:48.713 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:48.713 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.713 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:48.713 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:55:48.713 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:48.713 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.713 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.713 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:48.713 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:55:48.713 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:48.713 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:48.713 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:48.713 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.713 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:55:48.714 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:55:48.714 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:48.714 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:48.714 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:48.714 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.714 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.714 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:55:48.714 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:55:48.714 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:55:48.714 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:55:48.714 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.714 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:55:48.714 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:55:48.714 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:48.714 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.714 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.715 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:55:48.715 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:55:48.715 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:55:48.715 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:48.715 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:48.715 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.715 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:55:48.715 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:55:48.715 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:55:48.715 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:55:48.715 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:55:48.715 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:48.715 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:48.715 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.715 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:55:48.715 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:55:48.716 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.716 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:48.716 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:48.716 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:55:48.716 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:48.716 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:48.716 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:55:48.716 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.716 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:48.716 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:48.716 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2023-12-25T20:55:48.716 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:48.716 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:48.716 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:55:48.716 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.717 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:48.717 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:48.717 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2023-12-25T20:55:48.717 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:48.717 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:55:48.717 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2023-12-25T20:55:48.717 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:55:48.717 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:55:48.717 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:55:48.717 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:55:48.717 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:55:48.717 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:55:48.717 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:55:48.717 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:55:48.717 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:55:48.719 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-25T20:55:48.719 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-12-25T20:55:49.210 INFO:tasks.workunit.client.0.smithi191.stderr:already renamed to 'bar-asdf' 2023-12-25T20:55:49.219 INFO:tasks.workunit.client.0.smithi191.stderr:+ ceph osd crush rule ls 2023-12-25T20:55:49.219 INFO:tasks.workunit.client.0.smithi191.stderr:+ grep foo-asdf 2023-12-25T20:55:49.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T20:55:49.611 INFO:tasks.workunit:Stopping ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh'] on client.0... 2023-12-25T20:55:49.611 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-25T20:55:49.862 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/workunit.py", line 423, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test mon/crush_ops.sh) on smithi191 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh' 2023-12-25T20:55:49.946 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d84aeaee14174e399a73444e7021da86 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/workunit.py", line 423, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test mon/crush_ops.sh) on smithi191 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh' 2023-12-25T20:55:49.949 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2023-12-25T20:55:49.958 INFO:tasks.mon_thrash:joining mon_thrasher 2023-12-25T20:56:08.719 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-12-25T20:56:08.720 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2023-12-25T20:56:08.720 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-25T20:56:08.720 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-12-25T20:56:08.723 INFO:tasks.ceph.mon.b:Started 2023-12-25T20:56:08.724 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-25T20:56:08.724 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:56:09.091 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":44,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:56:12.105 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:56:12.475 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":44,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:56:15.489 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:56:15.855 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:56:15.868 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-25T20:56:15.868 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-25T20:56:15.869 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-25T20:56:16.005 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:56:16.005 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:56:16.005 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:56:16.005 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2023-12-25T20:56:16.006 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 46, 2023-12-25T20:56:16.006 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:56:16.006 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:56:16.006 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:56:16.006 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:56:16.006 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:56:16.006 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2023-12-25T20:56:16.006 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:56:16.006 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:56:16.007 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:56:16.007 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:56:16.007 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:56:16.007 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:56:16.007 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:56:16.007 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:56:16.007 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:56:16.007 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:56:16.007 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:56:16.008 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:56:16.008 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:56:16.008 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:56:16.008 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:56:16.008 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:56:16.008 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:56:16.008 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:56:16.008 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:56:16.008 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:56:16.009 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:56:16.009 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:56:16.009 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:56:16.009 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.009 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:56:16.009 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:56:16.009 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:56:16.009 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:56:16.010 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:56:16.010 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:56:16.010 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:56:16.010 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:56:16.010 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:56:16.010 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:56:16.010 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:56:16.010 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:56:16.010 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:56:16.011 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:56:16.011 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:56:16.011 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:56:16.011 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:56:16.011 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:56:16.011 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:56:16.011 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:56:16.011 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:56:16.011 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:56:16.011 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:56:16.012 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:56:16.012 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:56:16.012 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:56:16.012 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:56:16.012 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.012 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:56:16.012 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.012 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:56:16.013 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:56:16.013 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:56:16.013 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:56:16.013 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.013 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:56:16.013 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:56:16.013 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:56:16.013 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.013 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.014 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:56:16.014 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:56:16.014 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:56:16.014 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:56:16.014 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:56:16.014 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.014 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:56:16.014 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:56:16.014 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:56:16.015 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:56:16.015 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:56:16.015 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.015 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.015 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:56:16.015 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:56:16.015 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:56:16.015 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:56:16.015 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.016 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:56:16.016 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:56:16.016 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:56:16.016 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.016 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.016 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:56:16.016 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:56:16.016 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:56:16.016 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:56:16.017 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:56:16.017 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.017 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:56:16.017 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:56:16.017 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:56:16.017 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:56:16.017 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:56:16.017 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.018 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.018 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:56:16.018 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:56:16.018 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:56:16.018 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:56:16.018 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.018 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:56:16.018 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:56:16.018 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:56:16.019 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.019 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.019 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:56:16.019 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:56:16.019 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:56:16.019 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:56:16.019 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:56:16.019 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.019 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:56:16.020 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:56:16.020 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:56:16.020 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:56:16.020 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:56:16.020 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:56:16.020 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:56:16.020 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.020 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:56:16.020 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:56:16.021 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.021 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:56:16.021 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:56:16.021 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:56:16.021 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:56:16.021 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:56:16.021 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:56:16.021 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.021 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:56:16.022 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:56:16.022 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2023-12-25T20:56:16.022 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:56:16.022 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:56:16.022 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:56:16.022 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.022 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:56:16.022 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:56:16.022 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2023-12-25T20:56:16.023 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:56:16.023 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:56:16.023 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.023 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:56:16.023 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:56:16.024 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:56:16.024 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-25T20:56:16.025 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-25T20:56:16.162 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:56:16.162 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:56:16.163 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:56:16.163 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2023-12-25T20:56:16.163 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 46, 2023-12-25T20:56:16.163 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:56:16.163 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:56:16.163 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:56:16.163 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:56:16.163 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:56:16.164 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2023-12-25T20:56:16.164 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:56:16.164 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:56:16.164 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:56:16.164 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:56:16.164 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:56:16.164 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:56:16.164 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:56:16.164 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:56:16.165 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:56:16.165 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:56:16.165 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:56:16.165 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:56:16.165 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:56:16.165 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:56:16.165 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:56:16.165 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:56:16.165 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:56:16.166 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:56:16.166 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:56:16.166 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:56:16.166 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:56:16.166 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:56:16.166 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:56:16.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.166 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:56:16.167 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:56:16.167 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:56:16.167 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:56:16.167 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:56:16.167 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:56:16.167 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:56:16.167 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:56:16.167 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:56:16.167 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:56:16.168 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:56:16.168 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:56:16.168 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:56:16.168 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:56:16.168 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:56:16.168 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:56:16.168 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:56:16.168 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:56:16.168 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:56:16.168 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:56:16.169 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:56:16.169 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:56:16.169 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:56:16.169 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:56:16.169 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:56:16.169 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:56:16.169 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:56:16.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.170 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:56:16.170 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.170 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:56:16.170 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:56:16.170 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:56:16.170 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:56:16.170 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.170 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:56:16.170 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:56:16.171 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:56:16.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.171 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.171 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:56:16.171 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:56:16.171 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:56:16.171 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:56:16.171 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:56:16.172 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.172 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:56:16.172 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:56:16.172 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:56:16.172 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:56:16.172 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:56:16.172 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.172 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.172 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:56:16.173 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:56:16.173 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:56:16.173 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:56:16.173 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.173 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:56:16.173 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:56:16.173 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:56:16.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.173 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.174 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:56:16.174 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:56:16.174 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:56:16.174 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:56:16.174 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:56:16.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.174 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:56:16.174 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:56:16.174 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:56:16.175 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:56:16.175 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:56:16.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.175 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.175 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:56:16.175 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:56:16.175 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:56:16.175 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:56:16.175 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.176 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:56:16.176 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:56:16.176 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:56:16.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.176 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.176 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:56:16.176 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:56:16.176 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:56:16.176 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:56:16.177 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:56:16.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.177 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:56:16.177 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:56:16.177 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:56:16.177 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:56:16.177 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:56:16.177 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:56:16.177 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:56:16.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.178 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:56:16.178 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:56:16.178 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.178 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:56:16.178 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:56:16.178 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:56:16.178 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:56:16.178 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:56:16.178 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2023-12-25T20:56:16.179 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.179 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:56:16.179 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:56:16.179 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2023-12-25T20:56:16.179 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:56:16.179 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:56:16.179 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:56:16.179 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.180 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:56:16.180 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:56:16.180 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2023-12-25T20:56:16.180 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:56:16.180 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:56:16.180 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2023-12-25T20:56:16.180 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:16.181 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:56:16.181 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:56:16.181 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:56:16.181 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:56:16.181 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:56:16.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:16.182 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:56:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:56:16.183 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:56:16.183 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-25T20:56:16.183 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-25T20:56:20.319 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:56:20.319 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:56:20.320 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:56:20.320 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2023-12-25T20:56:20.320 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 46, 2023-12-25T20:56:20.320 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2023-12-25T20:56:20.320 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2023-12-25T20:56:20.320 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2023-12-25T20:56:20.320 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2023-12-25T20:56:20.320 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:56:20.321 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 6, 2023-12-25T20:56:20.321 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:56:20.321 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958747317026820", 2023-12-25T20:56:20.321 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2023-12-25T20:56:20.321 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:56:20.321 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:56:20.321 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:56:20.321 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:56:20.321 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:56:20.322 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:56:20.322 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:56:20.322 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:56:20.322 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:56:20.322 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138314316775423", 2023-12-25T20:56:20.322 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2023-12-25T20:56:20.322 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:56:20.322 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:56:20.323 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:56:20.323 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:56:20.323 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:56:20.323 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:56:20.323 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:56:20.323 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:56:20.323 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:56:20.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:20.323 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2023-12-25T20:56:20.324 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2023-12-25T20:56:20.324 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2023-12-25T20:56:20.324 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2023-12-25T20:56:20.324 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2023-12-25T20:56:20.324 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0ece9abc-d1cb-4514-a255-e770deeb21b9", 2023-12-25T20:56:20.324 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:56:20.324 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-12-25T20:51:52.261713Z", 2023-12-25T20:56:20.324 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 16, 2023-12-25T20:56:20.324 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "pacific", 2023-12-25T20:56:20.325 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2023-12-25T20:56:20.325 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2023-12-25T20:56:20.325 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2023-12-25T20:56:20.325 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2023-12-25T20:56:20.325 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2023-12-25T20:56:20.325 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2023-12-25T20:56:20.325 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2023-12-25T20:56:20.325 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2023-12-25T20:56:20.325 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2023-12-25T20:56:20.326 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2023-12-25T20:56:20.326 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2023-12-25T20:56:20.326 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2023-12-25T20:56:20.326 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2023-12-25T20:56:20.326 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2023-12-25T20:56:20.326 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging" 2023-12-25T20:56:20.326 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:56:20.327 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2023-12-25T20:56:20.327 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:20.327 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2023-12-25T20:56:20.327 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:20.327 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2023-12-25T20:56:20.327 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2023-12-25T20:56:20.327 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:56:20.328 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:56:20.328 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:20.328 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:56:20.328 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2023-12-25T20:56:20.328 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:56:20.328 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:20.328 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:20.328 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:56:20.328 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2023-12-25T20:56:20.328 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:56:20.329 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:56:20.329 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:56:20.329 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:20.329 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2023-12-25T20:56:20.329 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2023-12-25T20:56:20.329 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:56:20.329 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:56:20.329 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:56:20.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:20.330 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:20.330 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2023-12-25T20:56:20.330 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2023-12-25T20:56:20.330 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:56:20.330 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:56:20.330 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:20.330 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:56:20.330 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:3300", 2023-12-25T20:56:20.331 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:56:20.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:20.331 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:20.331 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:56:20.331 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789", 2023-12-25T20:56:20.331 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:56:20.332 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:56:20.332 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:56:20.332 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:20.332 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.191:6789/0", 2023-12-25T20:56:20.332 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.191:6789/0", 2023-12-25T20:56:20.333 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:56:20.333 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:56:20.333 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:56:20.334 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:20.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:20.334 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2023-12-25T20:56:20.334 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2023-12-25T20:56:20.334 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-12-25T20:56:20.334 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-12-25T20:56:20.335 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:20.335 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-12-25T20:56:20.335 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2023-12-25T20:56:20.335 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:56:20.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:20.335 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:20.336 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-12-25T20:56:20.336 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2023-12-25T20:56:20.336 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2023-12-25T20:56:20.336 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:56:20.336 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:56:20.336 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:20.337 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2023-12-25T20:56:20.337 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2023-12-25T20:56:20.337 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2023-12-25T20:56:20.337 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2023-12-25T20:56:20.338 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2023-12-25T20:56:20.338 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:56:20.338 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:56:20.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:20.338 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2023-12-25T20:56:20.338 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2023-12-25T20:56:20.338 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:20.339 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:56:20.339 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:56:20.339 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:56:20.339 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:56:20.339 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-12-25T20:56:20.339 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2023-12-25T20:56:20.340 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-12-25T20:56:20.340 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-25T20:56:20.340 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2023-12-25T20:56:20.340 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2023-12-25T20:56:20.340 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-12-25T20:56:20.340 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-12-25T20:56:20.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-12-25T20:56:20.340 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2023-12-25T20:56:20.340 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:56:20.342 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-25T20:56:20.342 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-12-25T20:56:20.342 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-12-25T20:56:20.706 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.706+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:56:20.706 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.706+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=12,kv=6,logm=67} crc {auth=1066711170,config=3140514009,health=3619357427,kv=661943862,logm=3349385003}) 2023-12-25T20:56:20.706 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.706+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=2,health=12,kv=6,logm=67} crc {auth=1066711170,config=3140514009,health=257751040,kv=661943861,logm=749566231}) 2023-12-25T20:56:20.706 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.706+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:56:20.706 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.706+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=12,kv=6,logm=67} crc {auth=1066711170,config=3140514009,health=3619357427,kv=661943862,logm=3349385003}) 2023-12-25T20:56:20.707 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.706+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=11,kv=6,logm=69} crc {auth=3146102264,config=3140514009,health=821451470,kv=661943861,logm=1764839490}) 2023-12-25T20:56:20.708 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.708+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:56:20.708 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.708+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1816091464}) 2023-12-25T20:56:20.709 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.708+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3947559007}) 2023-12-25T20:56:20.709 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.708+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:56:20.709 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.709+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1816091464}) 2023-12-25T20:56:20.709 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.709+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2372058933}) 2023-12-25T20:56:20.711 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.711+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:56:20.711 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.711+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=49,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=3,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=17} crc {logm=2900164,mdsmap=1531151339,mgr=3694473846,mgr_command_descs=1448462394,mgr_metadata=1482708640,mgrstat=330695580,mon_config_key=2934451455,monmap=3296645973,osd_metadata=3245185047,osd_pg_creating=1725079635,osdmap=1829950358}) 2023-12-25T20:56:20.711 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.711+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=49,mdsmap=3,mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=16} crc {logm=3935256437,mdsmap=1531151340,mgr=1272337822,mgr_command_descs=1448462394,mgr_metadata=1482708640,mgrstat=1616697891,mon_config_key=2934451455,monmap=3296645973,osd_metadata=1683759159,osd_pg_creating=1725079635,osdmap=2588471473}) 2023-12-25T20:56:20.711 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.711+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:56:20.711 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.711+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=49,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=3,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=17} crc {logm=2900164,mdsmap=1531151339,mgr=3694473846,mgr_command_descs=1448462394,mgr_metadata=1482708640,mgrstat=330695580,mon_config_key=2934451455,monmap=3296645973,osd_metadata=3245185047,osd_pg_creating=1725079635,osdmap=1829950358}) 2023-12-25T20:56:20.711 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.711+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=49,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=2,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=15} crc {logm=1790134440,mdsmap=1531151339,mgr=2949565714,mgr_command_descs=1448462394,mgr_metadata=1482708640,mgrstat=1616697891,mon_config_key=1642228043,monmap=3296645973,osd_metadata=1683759159,osd_pg_creating=1725079635,osdmap=2520622172}) 2023-12-25T20:56:20.715 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.715+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:56:20.715 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.715+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=47} crc {osdmap=4263548906}) 2023-12-25T20:56:20.715 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.715+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=49} crc {osdmap=63696880}) 2023-12-25T20:56:20.715 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.715+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-25T20:56:20.715 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.715+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=47} crc {osdmap=4263548906}) 2023-12-25T20:56:20.716 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-12-25T20:56:20.715+0000 7fdf6cd22700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=51} crc {osdmap=2695779619}) 2023-12-25T20:56:20.722 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-12-25T20:56:21.723 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-25T20:56:21.724 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-25T20:56:22.086 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","modified":"2023-12-25T20:51:52.261713Z","created":"2023-12-25T20:51:52.261713Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-25T20:56:22.099 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-25T20:56:22.100 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-25T20:56:22.109 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-12-25T20:56:22.109 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-25T20:56:22.401 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:56:22.402 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":63,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","created":"2023-12-25T20:52:22.419978+0000","modified":"2023-12-25T20:55:49.080668+0000","last_up_change":"2023-12-25T20:52:31.268017+0000","last_in_change":"2023-12-25T20:52:24.636112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":21,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-25T20:52:31.304424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-25T20:52:32.915209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"595d263d-6a56-4ccc-bb15-cd8d72e49f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":114394},{"type":"v1","addr":"172.21.15.153:6801","nonce":114394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":114394},{"type":"v1","addr":"172.21.15.153:6803","nonce":114394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":114394},{"type":"v1","addr":"172.21.15.153:6807","nonce":114394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":114394},{"type":"v1","addr":"172.21.15.153:6805","nonce":114394}]},"public_addr":"172.21.15.153:6801/114394","cluster_addr":"172.21.15.153:6803/114394","heartbeat_back_addr":"172.21.15.153:6807/114394","heartbeat_front_addr":"172.21.15.153:6805/114394","state":["exists","up"]},{"osd":1,"uuid":"3b8105a7-7f13-49a6-9388-23a4d9d131e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":114395},{"type":"v1","addr":"172.21.15.153:6809","nonce":114395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":114395},{"type":"v1","addr":"172.21.15.153:6811","nonce":114395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":114395},{"type":"v1","addr":"172.21.15.153:6815","nonce":114395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":114395},{"type":"v1","addr":"172.21.15.153:6813","nonce":114395}]},"public_addr":"172.21.15.153:6809/114395","cluster_addr":"172.21.15.153:6811/114395","heartbeat_back_addr":"172.21.15.153:6815/114395","heartbeat_front_addr":"172.21.15.153:6813/114395","state":["exists","up"]},{"osd":2,"uuid":"6b62f25d-a6fa-4089-9fd1-1b08d2cbad57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":114396},{"type":"v1","addr":"172.21.15.153:6817","nonce":114396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":114396},{"type":"v1","addr":"172.21.15.153:6819","nonce":114396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":114396},{"type":"v1","addr":"172.21.15.153:6823","nonce":114396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":114396},{"type":"v1","addr":"172.21.15.153:6821","nonce":114396}]},"public_addr":"172.21.15.153:6817/114396","cluster_addr":"172.21.15.153:6819/114396","heartbeat_back_addr":"172.21.15.153:6823/114396","heartbeat_front_addr":"172.21.15.153:6821/114396","state":["exists","up"]},{"osd":3,"uuid":"751e8edf-7ff9-4b73-9e8d-38416db248a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":114428},{"type":"v1","addr":"172.21.15.191:6801","nonce":114428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":114428},{"type":"v1","addr":"172.21.15.191:6803","nonce":114428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":114428},{"type":"v1","addr":"172.21.15.191:6807","nonce":114428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":114428},{"type":"v1","addr":"172.21.15.191:6805","nonce":114428}]},"public_addr":"172.21.15.191:6801/114428","cluster_addr":"172.21.15.191:6803/114428","heartbeat_back_addr":"172.21.15.191:6807/114428","heartbeat_front_addr":"172.21.15.191:6805/114428","state":["exists","up"]},{"osd":4,"uuid":"7635bbb7-ef87-483c-aadd-1984f330944c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":114429},{"type":"v1","addr":"172.21.15.191:6809","nonce":114429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":114429},{"type":"v1","addr":"172.21.15.191:6811","nonce":114429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":114429},{"type":"v1","addr":"172.21.15.191:6815","nonce":114429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":114429},{"type":"v1","addr":"172.21.15.191:6813","nonce":114429}]},"public_addr":"172.21.15.191:6809/114429","cluster_addr":"172.21.15.191:6811/114429","heartbeat_back_addr":"172.21.15.191:6815/114429","heartbeat_front_addr":"172.21.15.191:6813/114429","state":["exists","up"]},{"osd":5,"uuid":"7519398e-167d-40be-847b-1bea65ef0ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":114430},{"type":"v1","addr":"172.21.15.191:6817","nonce":114430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":114430},{"type":"v1","addr":"172.21.15.191:6819","nonce":114430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":114430},{"type":"v1","addr":"172.21.15.191:6823","nonce":114430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":114430},{"type":"v1","addr":"172.21.15.191:6821","nonce":114430}]},"public_addr":"172.21.15.191:6817/114430","cluster_addr":"172.21.15.191:6819/114430","heartbeat_back_addr":"172.21.15.191:6823/114430","heartbeat_front_addr":"172.21.15.191:6821/114430","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.682269+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.746139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.820727+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.689025+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.837599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.746784+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"ec-foo-ssd":{"crush-device-class":"ssd","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-25T20:56:22.415 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-25T20:56:22.415 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-25T20:56:22.703 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:56:22.704 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-12-25T20:56:22.715 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2023-12-25T20:56:21.785835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786992,"kb_used_data":4416,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564444048,"statfs":{"total":579820584960,"available":577990705152,"internally_reserved":0,"allocated":4521984,"data_stored":1943586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:33.019119+0000","last_change":"2023-12-25T20:52:34.300748+0000","last_active":"2023-12-25T20:55:33.019119+0000","last_peered":"2023-12-25T20:55:33.019119+0000","last_clean":"2023-12-25T20:55:33.019119+0000","last_became_active":"2023-12-25T20:52:34.300369+0000","last_became_peered":"2023-12-25T20:52:34.300369+0000","last_unstale":"2023-12-25T20:55:33.019119+0000","last_undegraded":"2023-12-25T20:55:33.019119+0000","last_fullsized":"2023-12-25T20:55:33.019119+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:32.381663+0000","last_change":"2023-12-25T20:52:34.295842+0000","last_active":"2023-12-25T20:55:32.381663+0000","last_peered":"2023-12-25T20:55:32.381663+0000","last_clean":"2023-12-25T20:55:32.381663+0000","last_became_active":"2023-12-25T20:52:34.295559+0000","last_became_peered":"2023-12-25T20:52:34.295559+0000","last_unstale":"2023-12-25T20:55:32.381663+0000","last_undegraded":"2023-12-25T20:55:32.381663+0000","last_fullsized":"2023-12-25T20:55:32.381663+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:34.527729+0000","last_change":"2023-12-25T20:52:34.300668+0000","last_active":"2023-12-25T20:55:34.527729+0000","last_peered":"2023-12-25T20:55:34.527729+0000","last_clean":"2023-12-25T20:55:34.527729+0000","last_became_active":"2023-12-25T20:52:34.300344+0000","last_became_peered":"2023-12-25T20:52:34.300344+0000","last_unstale":"2023-12-25T20:55:34.527729+0000","last_undegraded":"2023-12-25T20:55:34.527729+0000","last_fullsized":"2023-12-25T20:55:34.527729+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:33.165253+0000","last_change":"2023-12-25T20:52:34.297698+0000","last_active":"2023-12-25T20:55:33.165253+0000","last_peered":"2023-12-25T20:55:33.165253+0000","last_clean":"2023-12-25T20:55:33.165253+0000","last_became_active":"2023-12-25T20:52:34.297376+0000","last_became_peered":"2023-12-25T20:52:34.297376+0000","last_unstale":"2023-12-25T20:55:33.165253+0000","last_undegraded":"2023-12-25T20:55:33.165253+0000","last_fullsized":"2023-12-25T20:55:33.165253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:33.165253+0000","last_change":"2023-12-25T20:52:33.421321+0000","last_active":"2023-12-25T20:55:33.165253+0000","last_peered":"2023-12-25T20:55:33.165253+0000","last_clean":"2023-12-25T20:55:33.165253+0000","last_became_active":"2023-12-25T20:52:33.421006+0000","last_became_peered":"2023-12-25T20:52:33.421006+0000","last_unstale":"2023-12-25T20:55:33.165253+0000","last_undegraded":"2023-12-25T20:55:33.165253+0000","last_fullsized":"2023-12-25T20:55:33.165253+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:32.273733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:32.273733+0000","last_clean_scrub_stamp":"2023-12-25T20:52:32.273733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":51,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:34.195918+0000","last_change":"2023-12-25T20:52:34.294456+0000","last_active":"2023-12-25T20:55:34.195918+0000","last_peered":"2023-12-25T20:55:34.195918+0000","last_clean":"2023-12-25T20:55:34.195918+0000","last_became_active":"2023-12-25T20:52:34.294123+0000","last_became_peered":"2023-12-25T20:52:34.294123+0000","last_unstale":"2023-12-25T20:55:34.195918+0000","last_undegraded":"2023-12-25T20:55:34.195918+0000","last_fullsized":"2023-12-25T20:55:34.195918+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:33.165253+0000","last_change":"2023-12-25T20:52:34.300891+0000","last_active":"2023-12-25T20:55:33.165253+0000","last_peered":"2023-12-25T20:55:33.165253+0000","last_clean":"2023-12-25T20:55:33.165253+0000","last_became_active":"2023-12-25T20:52:34.300607+0000","last_became_peered":"2023-12-25T20:52:34.300607+0000","last_unstale":"2023-12-25T20:55:33.165253+0000","last_undegraded":"2023-12-25T20:55:33.165253+0000","last_fullsized":"2023-12-25T20:55:33.165253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:32.386759+0000","last_change":"2023-12-25T20:52:34.294057+0000","last_active":"2023-12-25T20:55:32.386759+0000","last_peered":"2023-12-25T20:55:32.386759+0000","last_clean":"2023-12-25T20:55:32.386759+0000","last_became_active":"2023-12-25T20:52:34.293804+0000","last_became_peered":"2023-12-25T20:52:34.293804+0000","last_unstale":"2023-12-25T20:55:32.386759+0000","last_undegraded":"2023-12-25T20:55:32.386759+0000","last_fullsized":"2023-12-25T20:55:32.386759+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:33.165253+0000","last_change":"2023-12-25T20:52:34.297698+0000","last_active":"2023-12-25T20:55:33.165253+0000","last_peered":"2023-12-25T20:55:33.165253+0000","last_clean":"2023-12-25T20:55:33.165253+0000","last_became_active":"2023-12-25T20:52:34.297388+0000","last_became_peered":"2023-12-25T20:52:34.297388+0000","last_unstale":"2023-12-25T20:55:33.165253+0000","last_undegraded":"2023-12-25T20:55:33.165253+0000","last_fullsized":"2023-12-25T20:55:33.165253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640304,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:55:41 2023","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":0.74299999999999999,"15min":0.55900000000000005},"min":{"1min":0.42599999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":22.896999999999998,"5min":22.896999999999998,"15min":22.896999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.976,"5min":0.78800000000000003,"15min":0.58899999999999997},"min":{"1min":0.40699999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":25.178000000000001,"5min":25.178000000000001,"15min":25.178000000000001},"last":0.51000000000000001}]},{"osd":1,"last update":"Mon Dec 25 20:55:41 2023","interfaces":[{"interface":"back","average":{"1min":2.1520000000000001,"5min":0.81699999999999995,"15min":0.59099999999999997},"min":{"1min":0.33300000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":31.620999999999999,"5min":31.620999999999999,"15min":31.620999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":3.016,"5min":1.008,"15min":0.67800000000000005},"min":{"1min":0.38700000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":47.942999999999998,"5min":47.942999999999998,"15min":47.942999999999998},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon Dec 25 20:55:41 2023","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":0.78300000000000003,"15min":0.55400000000000005},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":31.036999999999999,"5min":31.036999999999999,"15min":31.036999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":2.181,"5min":0.80000000000000004,"15min":0.56799999999999995},"min":{"1min":0.42099999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":31.064,"5min":31.064,"15min":31.064},"last":0.499}]},{"osd":3,"last update":"Mon Dec 25 20:55:41 2023","interfaces":[{"interface":"back","average":{"1min":1.976,"5min":0.66800000000000004,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.249,"15min":0.249},"max":{"1min":30.975999999999999,"5min":30.975999999999999,"15min":30.975999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.6699999999999999,"5min":0.63,"15min":0.45300000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":24.087,"5min":24.087,"15min":24.087},"last":0.623}]},{"osd":4,"last update":"Mon Dec 25 20:55:41 2023","interfaces":[{"interface":"back","average":{"1min":1.7190000000000001,"5min":0.65100000000000002,"15min":0.47499999999999998},"min":{"1min":0.26400000000000001,"5min":0.253,"15min":0.253},"max":{"1min":25.238,"5min":25.238,"15min":25.238},"last":0.64800000000000002},{"interface":"front","average":{"1min":2.0739999999999998,"5min":0.75600000000000001,"15min":0.53300000000000003},"min":{"1min":0.32300000000000001,"5min":0.317,"15min":0.317},"max":{"1min":31.577000000000002,"5min":31.577000000000002,"15min":31.577000000000002},"last":0.60099999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640304,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:55:37 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48299999999999998,"15min":0.47499999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.498,"15min":0.48799999999999999},"min":{"1min":0.38200000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.50700000000000001}]},{"osd":1,"last update":"Mon Dec 25 20:55:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.51600000000000001,"15min":0.52200000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.88900000000000001,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47199999999999998,"15min":0.46200000000000002},"min":{"1min":0.33300000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.442}]},{"osd":2,"last update":"Mon Dec 25 20:55:37 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47399999999999998,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.65800000000000003,"5min":1.056,"15min":1.056},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.46200000000000002,"15min":0.44700000000000001},"min":{"1min":0.32400000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.47299999999999998}]},{"osd":3,"last update":"Mon Dec 25 20:55:37 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.375,"15min":0.38300000000000001},"min":{"1min":0.24199999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.72499999999999998,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.376,"5min":0.35799999999999998,"15min":0.34999999999999998},"min":{"1min":0.26200000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.379}]},{"osd":5,"last update":"Mon Dec 25 20:55:37 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.46200000000000002,"15min":0.496},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60099999999999998,"5min":2.9500000000000002,"15min":2.9500000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.48299999999999998,"15min":0.51700000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.597,"5min":3.0099999999999998,"15min":3.0099999999999998},"last":0.41099999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640304,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.46500000000000002,"15min":0.44700000000000001},"min":{"1min":0.309,"5min":0.245,"15min":0.245},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.45700000000000002,"15min":0.437},"min":{"1min":0.33000000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.45700000000000002}]},{"osd":1,"last update":"Mon Dec 25 20:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.45500000000000002,"15min":0.439},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.443,"15min":0.42899999999999999},"min":{"1min":0.315,"5min":0.31,"15min":0.31},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Dec 25 20:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.45900000000000002,"15min":0.44500000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45800000000000002,"15min":0.442},"min":{"1min":0.33000000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.41999999999999998}]},{"osd":4,"last update":"Mon Dec 25 20:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.34300000000000003,"15min":0.33300000000000002},"min":{"1min":0.22800000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39000000000000001,"15min":0.38200000000000001},"min":{"1min":0.307,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.29499999999999998}]},{"osd":5,"last update":"Mon Dec 25 20:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26000000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.38,"5min":0.36199999999999999,"15min":0.35799999999999998},"min":{"1min":0.25800000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.317}]}]},{"osd":2,"up_from":11,"seq":47244640305,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.314,"15min":0.312},"min":{"1min":0.14099999999999999,"5min":0.14099999999999999,"15min":0.14099999999999999},"max":{"1min":0.41999999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.247},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.33800000000000002,"15min":0.33200000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.48399999999999999,"5min":0.495,"15min":0.495},"last":0.27200000000000002}]},{"osd":1,"last update":"Mon Dec 25 20:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.34999999999999998,"15min":0.35199999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.43099999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.33100000000000002,"15min":0.32700000000000001},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.38200000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.25700000000000001}]},{"osd":3,"last update":"Mon Dec 25 20:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.46400000000000002,"15min":0.44400000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.70199999999999996,"5min":0.748,"15min":0.748},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.46999999999999997,"15min":0.44900000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.379}]},{"osd":4,"last update":"Mon Dec 25 20:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.47199999999999998,"15min":0.44800000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.76300000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.498,"5min":0.48499999999999999,"15min":0.47299999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.66000000000000003,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.42599999999999999}]},{"osd":5,"last update":"Mon Dec 25 20:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.49399999999999999,"15min":0.48199999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65800000000000003,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.495,"15min":0.47699999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.73699999999999999,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.39300000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640304,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.309},"min":{"1min":0.223,"5min":0.186,"15min":0.186},"max":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"last":0.313},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.33700000000000002,"15min":0.33500000000000002},"min":{"1min":0.254,"5min":0.216,"15min":0.216},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.378}]},{"osd":2,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.34300000000000003,"15min":0.34399999999999997},"min":{"1min":0.27000000000000002,"5min":0.216,"15min":0.216},"max":{"1min":0.41499999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35599999999999998,"15min":0.35199999999999998},"min":{"1min":0.249,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.45100000000000001,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.34100000000000003}]},{"osd":3,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.46800000000000003,"15min":0.44400000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.504,"5min":0.47699999999999998,"15min":0.46400000000000002},"min":{"1min":0.35399999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.64600000000000002,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.51700000000000002}]},{"osd":4,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51500000000000001,"15min":0.49299999999999999},"min":{"1min":0.376,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.502},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.51900000000000002,"15min":0.498},"min":{"1min":0.38200000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68300000000000005,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.49099999999999999,"15min":0.47899999999999998},"min":{"1min":0.38800000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.51100000000000001,"15min":0.49099999999999999},"min":{"1min":0.38400000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.54600000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640305,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:55:33 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.32700000000000001,"15min":0.32100000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.313},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33500000000000002,"15min":0.33200000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34899999999999998,"15min":0.34999999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.496,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.36199999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.52800000000000002,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.34100000000000003}]},{"osd":3,"last update":"Mon Dec 25 20:55:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.47599999999999998,"15min":0.46200000000000002},"min":{"1min":0.33800000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.61399999999999999,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.505,"5min":0.502,"15min":0.48899999999999999},"min":{"1min":0.39100000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.60599999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.58099999999999996}]},{"osd":4,"last update":"Mon Dec 25 20:55:33 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47499999999999998,"15min":0.46400000000000002},"min":{"1min":0.34999999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.60299999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46700000000000003,"15min":0.45700000000000002},"min":{"1min":0.36499999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.54400000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.59899999999999998}]},{"osd":5,"last update":"Mon Dec 25 20:55:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.47699999999999998,"15min":0.46500000000000002},"min":{"1min":0.374,"5min":0.31,"15min":0.31},"max":{"1min":0.622,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.5,"15min":0.48999999999999999},"min":{"1min":0.38500000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.64500000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.65700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-25T20:56:22.716 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-25T20:56:23.004 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:56:23.006 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-12-25T20:56:23.017 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2023-12-25T20:56:21.785835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786992,"kb_used_data":4416,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564444048,"statfs":{"total":579820584960,"available":577990705152,"internally_reserved":0,"allocated":4521984,"data_stored":1943586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:33.019119+0000","last_change":"2023-12-25T20:52:34.300748+0000","last_active":"2023-12-25T20:55:33.019119+0000","last_peered":"2023-12-25T20:55:33.019119+0000","last_clean":"2023-12-25T20:55:33.019119+0000","last_became_active":"2023-12-25T20:52:34.300369+0000","last_became_peered":"2023-12-25T20:52:34.300369+0000","last_unstale":"2023-12-25T20:55:33.019119+0000","last_undegraded":"2023-12-25T20:55:33.019119+0000","last_fullsized":"2023-12-25T20:55:33.019119+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:32.381663+0000","last_change":"2023-12-25T20:52:34.295842+0000","last_active":"2023-12-25T20:55:32.381663+0000","last_peered":"2023-12-25T20:55:32.381663+0000","last_clean":"2023-12-25T20:55:32.381663+0000","last_became_active":"2023-12-25T20:52:34.295559+0000","last_became_peered":"2023-12-25T20:52:34.295559+0000","last_unstale":"2023-12-25T20:55:32.381663+0000","last_undegraded":"2023-12-25T20:55:32.381663+0000","last_fullsized":"2023-12-25T20:55:32.381663+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:34.527729+0000","last_change":"2023-12-25T20:52:34.300668+0000","last_active":"2023-12-25T20:55:34.527729+0000","last_peered":"2023-12-25T20:55:34.527729+0000","last_clean":"2023-12-25T20:55:34.527729+0000","last_became_active":"2023-12-25T20:52:34.300344+0000","last_became_peered":"2023-12-25T20:52:34.300344+0000","last_unstale":"2023-12-25T20:55:34.527729+0000","last_undegraded":"2023-12-25T20:55:34.527729+0000","last_fullsized":"2023-12-25T20:55:34.527729+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:33.165253+0000","last_change":"2023-12-25T20:52:34.297698+0000","last_active":"2023-12-25T20:55:33.165253+0000","last_peered":"2023-12-25T20:55:33.165253+0000","last_clean":"2023-12-25T20:55:33.165253+0000","last_became_active":"2023-12-25T20:52:34.297376+0000","last_became_peered":"2023-12-25T20:52:34.297376+0000","last_unstale":"2023-12-25T20:55:33.165253+0000","last_undegraded":"2023-12-25T20:55:33.165253+0000","last_fullsized":"2023-12-25T20:55:33.165253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:33.165253+0000","last_change":"2023-12-25T20:52:33.421321+0000","last_active":"2023-12-25T20:55:33.165253+0000","last_peered":"2023-12-25T20:55:33.165253+0000","last_clean":"2023-12-25T20:55:33.165253+0000","last_became_active":"2023-12-25T20:52:33.421006+0000","last_became_peered":"2023-12-25T20:52:33.421006+0000","last_unstale":"2023-12-25T20:55:33.165253+0000","last_undegraded":"2023-12-25T20:55:33.165253+0000","last_fullsized":"2023-12-25T20:55:33.165253+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:32.273733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:32.273733+0000","last_clean_scrub_stamp":"2023-12-25T20:52:32.273733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":51,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:34.195918+0000","last_change":"2023-12-25T20:52:34.294456+0000","last_active":"2023-12-25T20:55:34.195918+0000","last_peered":"2023-12-25T20:55:34.195918+0000","last_clean":"2023-12-25T20:55:34.195918+0000","last_became_active":"2023-12-25T20:52:34.294123+0000","last_became_peered":"2023-12-25T20:52:34.294123+0000","last_unstale":"2023-12-25T20:55:34.195918+0000","last_undegraded":"2023-12-25T20:55:34.195918+0000","last_fullsized":"2023-12-25T20:55:34.195918+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:33.165253+0000","last_change":"2023-12-25T20:52:34.300891+0000","last_active":"2023-12-25T20:55:33.165253+0000","last_peered":"2023-12-25T20:55:33.165253+0000","last_clean":"2023-12-25T20:55:33.165253+0000","last_became_active":"2023-12-25T20:52:34.300607+0000","last_became_peered":"2023-12-25T20:52:34.300607+0000","last_unstale":"2023-12-25T20:55:33.165253+0000","last_undegraded":"2023-12-25T20:55:33.165253+0000","last_fullsized":"2023-12-25T20:55:33.165253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:32.386759+0000","last_change":"2023-12-25T20:52:34.294057+0000","last_active":"2023-12-25T20:55:32.386759+0000","last_peered":"2023-12-25T20:55:32.386759+0000","last_clean":"2023-12-25T20:55:32.386759+0000","last_became_active":"2023-12-25T20:52:34.293804+0000","last_became_peered":"2023-12-25T20:52:34.293804+0000","last_unstale":"2023-12-25T20:55:32.386759+0000","last_undegraded":"2023-12-25T20:55:32.386759+0000","last_fullsized":"2023-12-25T20:55:32.386759+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:33.165253+0000","last_change":"2023-12-25T20:52:34.297698+0000","last_active":"2023-12-25T20:55:33.165253+0000","last_peered":"2023-12-25T20:55:33.165253+0000","last_clean":"2023-12-25T20:55:33.165253+0000","last_became_active":"2023-12-25T20:52:34.297388+0000","last_became_peered":"2023-12-25T20:52:34.297388+0000","last_unstale":"2023-12-25T20:55:33.165253+0000","last_undegraded":"2023-12-25T20:55:33.165253+0000","last_fullsized":"2023-12-25T20:55:33.165253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640304,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:55:41 2023","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":0.74299999999999999,"15min":0.55900000000000005},"min":{"1min":0.42599999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":22.896999999999998,"5min":22.896999999999998,"15min":22.896999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.976,"5min":0.78800000000000003,"15min":0.58899999999999997},"min":{"1min":0.40699999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":25.178000000000001,"5min":25.178000000000001,"15min":25.178000000000001},"last":0.51000000000000001}]},{"osd":1,"last update":"Mon Dec 25 20:55:41 2023","interfaces":[{"interface":"back","average":{"1min":2.1520000000000001,"5min":0.81699999999999995,"15min":0.59099999999999997},"min":{"1min":0.33300000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":31.620999999999999,"5min":31.620999999999999,"15min":31.620999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":3.016,"5min":1.008,"15min":0.67800000000000005},"min":{"1min":0.38700000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":47.942999999999998,"5min":47.942999999999998,"15min":47.942999999999998},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon Dec 25 20:55:41 2023","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":0.78300000000000003,"15min":0.55400000000000005},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":31.036999999999999,"5min":31.036999999999999,"15min":31.036999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":2.181,"5min":0.80000000000000004,"15min":0.56799999999999995},"min":{"1min":0.42099999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":31.064,"5min":31.064,"15min":31.064},"last":0.499}]},{"osd":3,"last update":"Mon Dec 25 20:55:41 2023","interfaces":[{"interface":"back","average":{"1min":1.976,"5min":0.66800000000000004,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.249,"15min":0.249},"max":{"1min":30.975999999999999,"5min":30.975999999999999,"15min":30.975999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.6699999999999999,"5min":0.63,"15min":0.45300000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":24.087,"5min":24.087,"15min":24.087},"last":0.623}]},{"osd":4,"last update":"Mon Dec 25 20:55:41 2023","interfaces":[{"interface":"back","average":{"1min":1.7190000000000001,"5min":0.65100000000000002,"15min":0.47499999999999998},"min":{"1min":0.26400000000000001,"5min":0.253,"15min":0.253},"max":{"1min":25.238,"5min":25.238,"15min":25.238},"last":0.64800000000000002},{"interface":"front","average":{"1min":2.0739999999999998,"5min":0.75600000000000001,"15min":0.53300000000000003},"min":{"1min":0.32300000000000001,"5min":0.317,"15min":0.317},"max":{"1min":31.577000000000002,"5min":31.577000000000002,"15min":31.577000000000002},"last":0.60099999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640304,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:55:37 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48299999999999998,"15min":0.47499999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.498,"15min":0.48799999999999999},"min":{"1min":0.38200000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.50700000000000001}]},{"osd":1,"last update":"Mon Dec 25 20:55:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.51600000000000001,"15min":0.52200000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.88900000000000001,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47199999999999998,"15min":0.46200000000000002},"min":{"1min":0.33300000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.442}]},{"osd":2,"last update":"Mon Dec 25 20:55:37 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47399999999999998,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.65800000000000003,"5min":1.056,"15min":1.056},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.46200000000000002,"15min":0.44700000000000001},"min":{"1min":0.32400000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.47299999999999998}]},{"osd":3,"last update":"Mon Dec 25 20:55:37 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.375,"15min":0.38300000000000001},"min":{"1min":0.24199999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.72499999999999998,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.376,"5min":0.35799999999999998,"15min":0.34999999999999998},"min":{"1min":0.26200000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.379}]},{"osd":5,"last update":"Mon Dec 25 20:55:37 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.46200000000000002,"15min":0.496},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60099999999999998,"5min":2.9500000000000002,"15min":2.9500000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.48299999999999998,"15min":0.51700000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.597,"5min":3.0099999999999998,"15min":3.0099999999999998},"last":0.41099999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640304,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.46500000000000002,"15min":0.44700000000000001},"min":{"1min":0.309,"5min":0.245,"15min":0.245},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.45700000000000002,"15min":0.437},"min":{"1min":0.33000000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.45700000000000002}]},{"osd":1,"last update":"Mon Dec 25 20:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.45500000000000002,"15min":0.439},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.443,"15min":0.42899999999999999},"min":{"1min":0.315,"5min":0.31,"15min":0.31},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Dec 25 20:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.45900000000000002,"15min":0.44500000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45800000000000002,"15min":0.442},"min":{"1min":0.33000000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.41999999999999998}]},{"osd":4,"last update":"Mon Dec 25 20:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.34300000000000003,"15min":0.33300000000000002},"min":{"1min":0.22800000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39000000000000001,"15min":0.38200000000000001},"min":{"1min":0.307,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.29499999999999998}]},{"osd":5,"last update":"Mon Dec 25 20:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26000000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.38,"5min":0.36199999999999999,"15min":0.35799999999999998},"min":{"1min":0.25800000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.317}]}]},{"osd":2,"up_from":11,"seq":47244640305,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.314,"15min":0.312},"min":{"1min":0.14099999999999999,"5min":0.14099999999999999,"15min":0.14099999999999999},"max":{"1min":0.41999999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.247},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.33800000000000002,"15min":0.33200000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.48399999999999999,"5min":0.495,"15min":0.495},"last":0.27200000000000002}]},{"osd":1,"last update":"Mon Dec 25 20:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.34999999999999998,"15min":0.35199999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.43099999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.33100000000000002,"15min":0.32700000000000001},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.38200000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.25700000000000001}]},{"osd":3,"last update":"Mon Dec 25 20:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.46400000000000002,"15min":0.44400000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.70199999999999996,"5min":0.748,"15min":0.748},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.46999999999999997,"15min":0.44900000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.379}]},{"osd":4,"last update":"Mon Dec 25 20:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.47199999999999998,"15min":0.44800000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.76300000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.498,"5min":0.48499999999999999,"15min":0.47299999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.66000000000000003,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.42599999999999999}]},{"osd":5,"last update":"Mon Dec 25 20:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.49399999999999999,"15min":0.48199999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65800000000000003,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.495,"15min":0.47699999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.73699999999999999,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.39300000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640304,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.309},"min":{"1min":0.223,"5min":0.186,"15min":0.186},"max":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"last":0.313},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.33700000000000002,"15min":0.33500000000000002},"min":{"1min":0.254,"5min":0.216,"15min":0.216},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.378}]},{"osd":2,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.34300000000000003,"15min":0.34399999999999997},"min":{"1min":0.27000000000000002,"5min":0.216,"15min":0.216},"max":{"1min":0.41499999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35599999999999998,"15min":0.35199999999999998},"min":{"1min":0.249,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.45100000000000001,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.34100000000000003}]},{"osd":3,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.46800000000000003,"15min":0.44400000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.504,"5min":0.47699999999999998,"15min":0.46400000000000002},"min":{"1min":0.35399999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.64600000000000002,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.51700000000000002}]},{"osd":4,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51500000000000001,"15min":0.49299999999999999},"min":{"1min":0.376,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.502},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.51900000000000002,"15min":0.498},"min":{"1min":0.38200000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68300000000000005,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.49099999999999999,"15min":0.47899999999999998},"min":{"1min":0.38800000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.51100000000000001,"15min":0.49099999999999999},"min":{"1min":0.38400000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.54600000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640305,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:55:33 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.32700000000000001,"15min":0.32100000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.313},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33500000000000002,"15min":0.33200000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34899999999999998,"15min":0.34999999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.496,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.36199999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.52800000000000002,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.34100000000000003}]},{"osd":3,"last update":"Mon Dec 25 20:55:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.47599999999999998,"15min":0.46200000000000002},"min":{"1min":0.33800000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.61399999999999999,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.505,"5min":0.502,"15min":0.48899999999999999},"min":{"1min":0.39100000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.60599999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.58099999999999996}]},{"osd":4,"last update":"Mon Dec 25 20:55:33 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47499999999999998,"15min":0.46400000000000002},"min":{"1min":0.34999999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.60299999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46700000000000003,"15min":0.45700000000000002},"min":{"1min":0.36499999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.54400000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.59899999999999998}]},{"osd":5,"last update":"Mon Dec 25 20:55:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.47699999999999998,"15min":0.46500000000000002},"min":{"1min":0.374,"5min":0.31,"15min":0.31},"max":{"1min":0.622,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.5,"15min":0.48999999999999999},"min":{"1min":0.38500000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.64500000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.65700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-25T20:56:23.018 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-25T20:56:23.018 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-25T20:56:23.308 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:56:23.309 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-12-25T20:56:23.320 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2023-12-25T20:56:21.785835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786992,"kb_used_data":4416,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564444048,"statfs":{"total":579820584960,"available":577990705152,"internally_reserved":0,"allocated":4521984,"data_stored":1943586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:33.019119+0000","last_change":"2023-12-25T20:52:34.300748+0000","last_active":"2023-12-25T20:55:33.019119+0000","last_peered":"2023-12-25T20:55:33.019119+0000","last_clean":"2023-12-25T20:55:33.019119+0000","last_became_active":"2023-12-25T20:52:34.300369+0000","last_became_peered":"2023-12-25T20:52:34.300369+0000","last_unstale":"2023-12-25T20:55:33.019119+0000","last_undegraded":"2023-12-25T20:55:33.019119+0000","last_fullsized":"2023-12-25T20:55:33.019119+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:32.381663+0000","last_change":"2023-12-25T20:52:34.295842+0000","last_active":"2023-12-25T20:55:32.381663+0000","last_peered":"2023-12-25T20:55:32.381663+0000","last_clean":"2023-12-25T20:55:32.381663+0000","last_became_active":"2023-12-25T20:52:34.295559+0000","last_became_peered":"2023-12-25T20:52:34.295559+0000","last_unstale":"2023-12-25T20:55:32.381663+0000","last_undegraded":"2023-12-25T20:55:32.381663+0000","last_fullsized":"2023-12-25T20:55:32.381663+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:34.527729+0000","last_change":"2023-12-25T20:52:34.300668+0000","last_active":"2023-12-25T20:55:34.527729+0000","last_peered":"2023-12-25T20:55:34.527729+0000","last_clean":"2023-12-25T20:55:34.527729+0000","last_became_active":"2023-12-25T20:52:34.300344+0000","last_became_peered":"2023-12-25T20:52:34.300344+0000","last_unstale":"2023-12-25T20:55:34.527729+0000","last_undegraded":"2023-12-25T20:55:34.527729+0000","last_fullsized":"2023-12-25T20:55:34.527729+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:33.165253+0000","last_change":"2023-12-25T20:52:34.297698+0000","last_active":"2023-12-25T20:55:33.165253+0000","last_peered":"2023-12-25T20:55:33.165253+0000","last_clean":"2023-12-25T20:55:33.165253+0000","last_became_active":"2023-12-25T20:52:34.297376+0000","last_became_peered":"2023-12-25T20:52:34.297376+0000","last_unstale":"2023-12-25T20:55:33.165253+0000","last_undegraded":"2023-12-25T20:55:33.165253+0000","last_fullsized":"2023-12-25T20:55:33.165253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:33.165253+0000","last_change":"2023-12-25T20:52:33.421321+0000","last_active":"2023-12-25T20:55:33.165253+0000","last_peered":"2023-12-25T20:55:33.165253+0000","last_clean":"2023-12-25T20:55:33.165253+0000","last_became_active":"2023-12-25T20:52:33.421006+0000","last_became_peered":"2023-12-25T20:52:33.421006+0000","last_unstale":"2023-12-25T20:55:33.165253+0000","last_undegraded":"2023-12-25T20:55:33.165253+0000","last_fullsized":"2023-12-25T20:55:33.165253+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:32.273733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:32.273733+0000","last_clean_scrub_stamp":"2023-12-25T20:52:32.273733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":51,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:34.195918+0000","last_change":"2023-12-25T20:52:34.294456+0000","last_active":"2023-12-25T20:55:34.195918+0000","last_peered":"2023-12-25T20:55:34.195918+0000","last_clean":"2023-12-25T20:55:34.195918+0000","last_became_active":"2023-12-25T20:52:34.294123+0000","last_became_peered":"2023-12-25T20:52:34.294123+0000","last_unstale":"2023-12-25T20:55:34.195918+0000","last_undegraded":"2023-12-25T20:55:34.195918+0000","last_fullsized":"2023-12-25T20:55:34.195918+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:33.165253+0000","last_change":"2023-12-25T20:52:34.300891+0000","last_active":"2023-12-25T20:55:33.165253+0000","last_peered":"2023-12-25T20:55:33.165253+0000","last_clean":"2023-12-25T20:55:33.165253+0000","last_became_active":"2023-12-25T20:52:34.300607+0000","last_became_peered":"2023-12-25T20:52:34.300607+0000","last_unstale":"2023-12-25T20:55:33.165253+0000","last_undegraded":"2023-12-25T20:55:33.165253+0000","last_fullsized":"2023-12-25T20:55:33.165253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:32.386759+0000","last_change":"2023-12-25T20:52:34.294057+0000","last_active":"2023-12-25T20:55:32.386759+0000","last_peered":"2023-12-25T20:55:32.386759+0000","last_clean":"2023-12-25T20:55:32.386759+0000","last_became_active":"2023-12-25T20:52:34.293804+0000","last_became_peered":"2023-12-25T20:52:34.293804+0000","last_unstale":"2023-12-25T20:55:32.386759+0000","last_undegraded":"2023-12-25T20:55:32.386759+0000","last_fullsized":"2023-12-25T20:55:32.386759+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:33.165253+0000","last_change":"2023-12-25T20:52:34.297698+0000","last_active":"2023-12-25T20:55:33.165253+0000","last_peered":"2023-12-25T20:55:33.165253+0000","last_clean":"2023-12-25T20:55:33.165253+0000","last_became_active":"2023-12-25T20:52:34.297388+0000","last_became_peered":"2023-12-25T20:52:34.297388+0000","last_unstale":"2023-12-25T20:55:33.165253+0000","last_undegraded":"2023-12-25T20:55:33.165253+0000","last_fullsized":"2023-12-25T20:55:33.165253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640304,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:55:41 2023","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":0.74299999999999999,"15min":0.55900000000000005},"min":{"1min":0.42599999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":22.896999999999998,"5min":22.896999999999998,"15min":22.896999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.976,"5min":0.78800000000000003,"15min":0.58899999999999997},"min":{"1min":0.40699999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":25.178000000000001,"5min":25.178000000000001,"15min":25.178000000000001},"last":0.51000000000000001}]},{"osd":1,"last update":"Mon Dec 25 20:55:41 2023","interfaces":[{"interface":"back","average":{"1min":2.1520000000000001,"5min":0.81699999999999995,"15min":0.59099999999999997},"min":{"1min":0.33300000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":31.620999999999999,"5min":31.620999999999999,"15min":31.620999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":3.016,"5min":1.008,"15min":0.67800000000000005},"min":{"1min":0.38700000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":47.942999999999998,"5min":47.942999999999998,"15min":47.942999999999998},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon Dec 25 20:55:41 2023","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":0.78300000000000003,"15min":0.55400000000000005},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":31.036999999999999,"5min":31.036999999999999,"15min":31.036999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":2.181,"5min":0.80000000000000004,"15min":0.56799999999999995},"min":{"1min":0.42099999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":31.064,"5min":31.064,"15min":31.064},"last":0.499}]},{"osd":3,"last update":"Mon Dec 25 20:55:41 2023","interfaces":[{"interface":"back","average":{"1min":1.976,"5min":0.66800000000000004,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.249,"15min":0.249},"max":{"1min":30.975999999999999,"5min":30.975999999999999,"15min":30.975999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.6699999999999999,"5min":0.63,"15min":0.45300000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":24.087,"5min":24.087,"15min":24.087},"last":0.623}]},{"osd":4,"last update":"Mon Dec 25 20:55:41 2023","interfaces":[{"interface":"back","average":{"1min":1.7190000000000001,"5min":0.65100000000000002,"15min":0.47499999999999998},"min":{"1min":0.26400000000000001,"5min":0.253,"15min":0.253},"max":{"1min":25.238,"5min":25.238,"15min":25.238},"last":0.64800000000000002},{"interface":"front","average":{"1min":2.0739999999999998,"5min":0.75600000000000001,"15min":0.53300000000000003},"min":{"1min":0.32300000000000001,"5min":0.317,"15min":0.317},"max":{"1min":31.577000000000002,"5min":31.577000000000002,"15min":31.577000000000002},"last":0.60099999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640304,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:55:37 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48299999999999998,"15min":0.47499999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.498,"15min":0.48799999999999999},"min":{"1min":0.38200000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.50700000000000001}]},{"osd":1,"last update":"Mon Dec 25 20:55:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.51600000000000001,"15min":0.52200000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.88900000000000001,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47199999999999998,"15min":0.46200000000000002},"min":{"1min":0.33300000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.442}]},{"osd":2,"last update":"Mon Dec 25 20:55:37 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47399999999999998,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.65800000000000003,"5min":1.056,"15min":1.056},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.46200000000000002,"15min":0.44700000000000001},"min":{"1min":0.32400000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.47299999999999998}]},{"osd":3,"last update":"Mon Dec 25 20:55:37 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.375,"15min":0.38300000000000001},"min":{"1min":0.24199999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.72499999999999998,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.376,"5min":0.35799999999999998,"15min":0.34999999999999998},"min":{"1min":0.26200000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.379}]},{"osd":5,"last update":"Mon Dec 25 20:55:37 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.46200000000000002,"15min":0.496},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60099999999999998,"5min":2.9500000000000002,"15min":2.9500000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.48299999999999998,"15min":0.51700000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.597,"5min":3.0099999999999998,"15min":3.0099999999999998},"last":0.41099999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640304,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.46500000000000002,"15min":0.44700000000000001},"min":{"1min":0.309,"5min":0.245,"15min":0.245},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.45700000000000002,"15min":0.437},"min":{"1min":0.33000000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.45700000000000002}]},{"osd":1,"last update":"Mon Dec 25 20:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.45500000000000002,"15min":0.439},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.443,"15min":0.42899999999999999},"min":{"1min":0.315,"5min":0.31,"15min":0.31},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Dec 25 20:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.45900000000000002,"15min":0.44500000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45800000000000002,"15min":0.442},"min":{"1min":0.33000000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.41999999999999998}]},{"osd":4,"last update":"Mon Dec 25 20:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.34300000000000003,"15min":0.33300000000000002},"min":{"1min":0.22800000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39000000000000001,"15min":0.38200000000000001},"min":{"1min":0.307,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.29499999999999998}]},{"osd":5,"last update":"Mon Dec 25 20:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26000000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.38,"5min":0.36199999999999999,"15min":0.35799999999999998},"min":{"1min":0.25800000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.317}]}]},{"osd":2,"up_from":11,"seq":47244640305,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.314,"15min":0.312},"min":{"1min":0.14099999999999999,"5min":0.14099999999999999,"15min":0.14099999999999999},"max":{"1min":0.41999999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.247},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.33800000000000002,"15min":0.33200000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.48399999999999999,"5min":0.495,"15min":0.495},"last":0.27200000000000002}]},{"osd":1,"last update":"Mon Dec 25 20:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.34999999999999998,"15min":0.35199999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.43099999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.33100000000000002,"15min":0.32700000000000001},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.38200000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.25700000000000001}]},{"osd":3,"last update":"Mon Dec 25 20:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.46400000000000002,"15min":0.44400000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.70199999999999996,"5min":0.748,"15min":0.748},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.46999999999999997,"15min":0.44900000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.379}]},{"osd":4,"last update":"Mon Dec 25 20:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.47199999999999998,"15min":0.44800000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.76300000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.498,"5min":0.48499999999999999,"15min":0.47299999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.66000000000000003,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.42599999999999999}]},{"osd":5,"last update":"Mon Dec 25 20:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.49399999999999999,"15min":0.48199999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65800000000000003,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.495,"15min":0.47699999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.73699999999999999,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.39300000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640304,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.309},"min":{"1min":0.223,"5min":0.186,"15min":0.186},"max":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"last":0.313},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.33700000000000002,"15min":0.33500000000000002},"min":{"1min":0.254,"5min":0.216,"15min":0.216},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.378}]},{"osd":2,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.34300000000000003,"15min":0.34399999999999997},"min":{"1min":0.27000000000000002,"5min":0.216,"15min":0.216},"max":{"1min":0.41499999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35599999999999998,"15min":0.35199999999999998},"min":{"1min":0.249,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.45100000000000001,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.34100000000000003}]},{"osd":3,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.46800000000000003,"15min":0.44400000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.504,"5min":0.47699999999999998,"15min":0.46400000000000002},"min":{"1min":0.35399999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.64600000000000002,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.51700000000000002}]},{"osd":4,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51500000000000001,"15min":0.49299999999999999},"min":{"1min":0.376,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.502},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.51900000000000002,"15min":0.498},"min":{"1min":0.38200000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68300000000000005,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.49099999999999999,"15min":0.47899999999999998},"min":{"1min":0.38800000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.51100000000000001,"15min":0.49099999999999999},"min":{"1min":0.38400000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.54600000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640305,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:55:33 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.32700000000000001,"15min":0.32100000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.313},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33500000000000002,"15min":0.33200000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34899999999999998,"15min":0.34999999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.496,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.36199999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.52800000000000002,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.34100000000000003}]},{"osd":3,"last update":"Mon Dec 25 20:55:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.47599999999999998,"15min":0.46200000000000002},"min":{"1min":0.33800000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.61399999999999999,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.505,"5min":0.502,"15min":0.48899999999999999},"min":{"1min":0.39100000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.60599999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.58099999999999996}]},{"osd":4,"last update":"Mon Dec 25 20:55:33 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47499999999999998,"15min":0.46400000000000002},"min":{"1min":0.34999999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.60299999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46700000000000003,"15min":0.45700000000000002},"min":{"1min":0.36499999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.54400000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.59899999999999998}]},{"osd":5,"last update":"Mon Dec 25 20:55:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.47699999999999998,"15min":0.46500000000000002},"min":{"1min":0.374,"5min":0.31,"15min":0.31},"max":{"1min":0.622,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.5,"15min":0.48999999999999999},"min":{"1min":0.38500000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.64500000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.65700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-25T20:56:23.322 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-25T20:56:23.614 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:56:23.615 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":63,"fsid":"0ece9abc-d1cb-4514-a255-e770deeb21b9","created":"2023-12-25T20:52:22.419978+0000","modified":"2023-12-25T20:55:49.080668+0000","last_up_change":"2023-12-25T20:52:31.268017+0000","last_in_change":"2023-12-25T20:52:24.636112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":21,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-25T20:52:31.304424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-25T20:52:32.915209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"595d263d-6a56-4ccc-bb15-cd8d72e49f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":114394},{"type":"v1","addr":"172.21.15.153:6801","nonce":114394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":114394},{"type":"v1","addr":"172.21.15.153:6803","nonce":114394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":114394},{"type":"v1","addr":"172.21.15.153:6807","nonce":114394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":114394},{"type":"v1","addr":"172.21.15.153:6805","nonce":114394}]},"public_addr":"172.21.15.153:6801/114394","cluster_addr":"172.21.15.153:6803/114394","heartbeat_back_addr":"172.21.15.153:6807/114394","heartbeat_front_addr":"172.21.15.153:6805/114394","state":["exists","up"]},{"osd":1,"uuid":"3b8105a7-7f13-49a6-9388-23a4d9d131e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":114395},{"type":"v1","addr":"172.21.15.153:6809","nonce":114395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":114395},{"type":"v1","addr":"172.21.15.153:6811","nonce":114395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":114395},{"type":"v1","addr":"172.21.15.153:6815","nonce":114395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":114395},{"type":"v1","addr":"172.21.15.153:6813","nonce":114395}]},"public_addr":"172.21.15.153:6809/114395","cluster_addr":"172.21.15.153:6811/114395","heartbeat_back_addr":"172.21.15.153:6815/114395","heartbeat_front_addr":"172.21.15.153:6813/114395","state":["exists","up"]},{"osd":2,"uuid":"6b62f25d-a6fa-4089-9fd1-1b08d2cbad57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":114396},{"type":"v1","addr":"172.21.15.153:6817","nonce":114396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":114396},{"type":"v1","addr":"172.21.15.153:6819","nonce":114396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":114396},{"type":"v1","addr":"172.21.15.153:6823","nonce":114396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":114396},{"type":"v1","addr":"172.21.15.153:6821","nonce":114396}]},"public_addr":"172.21.15.153:6817/114396","cluster_addr":"172.21.15.153:6819/114396","heartbeat_back_addr":"172.21.15.153:6823/114396","heartbeat_front_addr":"172.21.15.153:6821/114396","state":["exists","up"]},{"osd":3,"uuid":"751e8edf-7ff9-4b73-9e8d-38416db248a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":114428},{"type":"v1","addr":"172.21.15.191:6801","nonce":114428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":114428},{"type":"v1","addr":"172.21.15.191:6803","nonce":114428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":114428},{"type":"v1","addr":"172.21.15.191:6807","nonce":114428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":114428},{"type":"v1","addr":"172.21.15.191:6805","nonce":114428}]},"public_addr":"172.21.15.191:6801/114428","cluster_addr":"172.21.15.191:6803/114428","heartbeat_back_addr":"172.21.15.191:6807/114428","heartbeat_front_addr":"172.21.15.191:6805/114428","state":["exists","up"]},{"osd":4,"uuid":"7635bbb7-ef87-483c-aadd-1984f330944c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":114429},{"type":"v1","addr":"172.21.15.191:6809","nonce":114429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":114429},{"type":"v1","addr":"172.21.15.191:6811","nonce":114429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":114429},{"type":"v1","addr":"172.21.15.191:6815","nonce":114429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":114429},{"type":"v1","addr":"172.21.15.191:6813","nonce":114429}]},"public_addr":"172.21.15.191:6809/114429","cluster_addr":"172.21.15.191:6811/114429","heartbeat_back_addr":"172.21.15.191:6815/114429","heartbeat_front_addr":"172.21.15.191:6813/114429","state":["exists","up"]},{"osd":5,"uuid":"7519398e-167d-40be-847b-1bea65ef0ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":114430},{"type":"v1","addr":"172.21.15.191:6817","nonce":114430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":114430},{"type":"v1","addr":"172.21.15.191:6819","nonce":114430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":114430},{"type":"v1","addr":"172.21.15.191:6823","nonce":114430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":114430},{"type":"v1","addr":"172.21.15.191:6821","nonce":114430}]},"public_addr":"172.21.15.191:6817/114430","cluster_addr":"172.21.15.191:6819/114430","heartbeat_back_addr":"172.21.15.191:6823/114430","heartbeat_front_addr":"172.21.15.191:6821/114430","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.682269+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.746139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.820727+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.689025+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.837599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-25T20:52:29.746784+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"ec-foo-ssd":{"crush-device-class":"ssd","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-25T20:56:24.629 INFO:tasks.ceph:Scrubbing osd.0 2023-12-25T20:56:24.629 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-12-25T20:56:24.787 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:56:24.788 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-25T20:56:24.788 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:56:24.799 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-12-25T20:56:25.090 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 0 to deep-scrub 2023-12-25T20:56:25.100 INFO:tasks.ceph:Scrubbing osd.1 2023-12-25T20:56:25.100 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-12-25T20:56:25.248 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:56:25.248 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-25T20:56:25.248 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:56:25.260 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-12-25T20:56:25.557 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 1 to deep-scrub 2023-12-25T20:56:25.568 INFO:tasks.ceph:Scrubbing osd.2 2023-12-25T20:56:25.568 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-12-25T20:56:25.714 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:56:25.715 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-25T20:56:25.715 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:56:25.726 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-12-25T20:56:26.018 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 2 to deep-scrub 2023-12-25T20:56:26.029 INFO:tasks.ceph:Scrubbing osd.3 2023-12-25T20:56:26.030 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-12-25T20:56:26.179 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:56:26.179 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-25T20:56:26.179 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:56:26.191 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-12-25T20:56:26.493 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 3 to deep-scrub 2023-12-25T20:56:26.503 INFO:tasks.ceph:Scrubbing osd.4 2023-12-25T20:56:26.504 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-12-25T20:56:26.650 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:56:26.650 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-25T20:56:26.650 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:56:26.661 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-12-25T20:56:26.957 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 4 to deep-scrub 2023-12-25T20:56:26.968 INFO:tasks.ceph:Scrubbing osd.5 2023-12-25T20:56:26.968 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-12-25T20:56:27.115 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-12-25T20:56:27.115 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-25T20:56:27.115 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-12-25T20:56:27.126 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-12-25T20:56:27.418 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 5 to deep-scrub 2023-12-25T20:56:27.429 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-25T20:56:27.717 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:56:27.719 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-12-25T20:56:27.729 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2023-12-25T20:56:25.786631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786992,"kb_used_data":4416,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564444048,"statfs":{"total":579820584960,"available":577990705152,"internally_reserved":0,"allocated":4521984,"data_stored":1943586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:33.019119+0000","last_change":"2023-12-25T20:52:34.300748+0000","last_active":"2023-12-25T20:55:33.019119+0000","last_peered":"2023-12-25T20:55:33.019119+0000","last_clean":"2023-12-25T20:55:33.019119+0000","last_became_active":"2023-12-25T20:52:34.300369+0000","last_became_peered":"2023-12-25T20:52:34.300369+0000","last_unstale":"2023-12-25T20:55:33.019119+0000","last_undegraded":"2023-12-25T20:55:33.019119+0000","last_fullsized":"2023-12-25T20:55:33.019119+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:32.381663+0000","last_change":"2023-12-25T20:52:34.295842+0000","last_active":"2023-12-25T20:55:32.381663+0000","last_peered":"2023-12-25T20:55:32.381663+0000","last_clean":"2023-12-25T20:55:32.381663+0000","last_became_active":"2023-12-25T20:52:34.295559+0000","last_became_peered":"2023-12-25T20:52:34.295559+0000","last_unstale":"2023-12-25T20:55:32.381663+0000","last_undegraded":"2023-12-25T20:55:32.381663+0000","last_fullsized":"2023-12-25T20:55:32.381663+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:34.527729+0000","last_change":"2023-12-25T20:52:34.300668+0000","last_active":"2023-12-25T20:55:34.527729+0000","last_peered":"2023-12-25T20:55:34.527729+0000","last_clean":"2023-12-25T20:55:34.527729+0000","last_became_active":"2023-12-25T20:52:34.300344+0000","last_became_peered":"2023-12-25T20:52:34.300344+0000","last_unstale":"2023-12-25T20:55:34.527729+0000","last_undegraded":"2023-12-25T20:55:34.527729+0000","last_fullsized":"2023-12-25T20:55:34.527729+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:33.165253+0000","last_change":"2023-12-25T20:52:34.297698+0000","last_active":"2023-12-25T20:55:33.165253+0000","last_peered":"2023-12-25T20:55:33.165253+0000","last_clean":"2023-12-25T20:55:33.165253+0000","last_became_active":"2023-12-25T20:52:34.297376+0000","last_became_peered":"2023-12-25T20:52:34.297376+0000","last_unstale":"2023-12-25T20:55:33.165253+0000","last_undegraded":"2023-12-25T20:55:33.165253+0000","last_fullsized":"2023-12-25T20:55:33.165253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:33.165253+0000","last_change":"2023-12-25T20:52:33.421321+0000","last_active":"2023-12-25T20:55:33.165253+0000","last_peered":"2023-12-25T20:55:33.165253+0000","last_clean":"2023-12-25T20:55:33.165253+0000","last_became_active":"2023-12-25T20:52:33.421006+0000","last_became_peered":"2023-12-25T20:52:33.421006+0000","last_unstale":"2023-12-25T20:55:33.165253+0000","last_undegraded":"2023-12-25T20:55:33.165253+0000","last_fullsized":"2023-12-25T20:55:33.165253+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:32.273733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:32.273733+0000","last_clean_scrub_stamp":"2023-12-25T20:52:32.273733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":51,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:34.195918+0000","last_change":"2023-12-25T20:52:34.294456+0000","last_active":"2023-12-25T20:55:34.195918+0000","last_peered":"2023-12-25T20:55:34.195918+0000","last_clean":"2023-12-25T20:55:34.195918+0000","last_became_active":"2023-12-25T20:52:34.294123+0000","last_became_peered":"2023-12-25T20:52:34.294123+0000","last_unstale":"2023-12-25T20:55:34.195918+0000","last_undegraded":"2023-12-25T20:55:34.195918+0000","last_fullsized":"2023-12-25T20:55:34.195918+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:33.165253+0000","last_change":"2023-12-25T20:52:34.300891+0000","last_active":"2023-12-25T20:55:33.165253+0000","last_peered":"2023-12-25T20:55:33.165253+0000","last_clean":"2023-12-25T20:55:33.165253+0000","last_became_active":"2023-12-25T20:52:34.300607+0000","last_became_peered":"2023-12-25T20:52:34.300607+0000","last_unstale":"2023-12-25T20:55:33.165253+0000","last_undegraded":"2023-12-25T20:55:33.165253+0000","last_fullsized":"2023-12-25T20:55:33.165253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:32.386759+0000","last_change":"2023-12-25T20:52:34.294057+0000","last_active":"2023-12-25T20:55:32.386759+0000","last_peered":"2023-12-25T20:55:32.386759+0000","last_clean":"2023-12-25T20:55:32.386759+0000","last_became_active":"2023-12-25T20:52:34.293804+0000","last_became_peered":"2023-12-25T20:52:34.293804+0000","last_unstale":"2023-12-25T20:55:32.386759+0000","last_undegraded":"2023-12-25T20:55:32.386759+0000","last_fullsized":"2023-12-25T20:55:32.386759+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-25T20:55:33.165253+0000","last_change":"2023-12-25T20:52:34.297698+0000","last_active":"2023-12-25T20:55:33.165253+0000","last_peered":"2023-12-25T20:55:33.165253+0000","last_clean":"2023-12-25T20:55:33.165253+0000","last_became_active":"2023-12-25T20:52:34.297388+0000","last_became_peered":"2023-12-25T20:52:34.297388+0000","last_unstale":"2023-12-25T20:55:33.165253+0000","last_undegraded":"2023-12-25T20:55:33.165253+0000","last_fullsized":"2023-12-25T20:55:33.165253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:52:33.281268+0000","last_clean_scrub_stamp":"2023-12-25T20:52:33.281268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640305,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:55:41 2023","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":0.74299999999999999,"15min":0.55900000000000005},"min":{"1min":0.42599999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":22.896999999999998,"5min":22.896999999999998,"15min":22.896999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.976,"5min":0.78800000000000003,"15min":0.58899999999999997},"min":{"1min":0.40699999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":25.178000000000001,"5min":25.178000000000001,"15min":25.178000000000001},"last":0.51000000000000001}]},{"osd":1,"last update":"Mon Dec 25 20:55:41 2023","interfaces":[{"interface":"back","average":{"1min":2.1520000000000001,"5min":0.81699999999999995,"15min":0.59099999999999997},"min":{"1min":0.33300000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":31.620999999999999,"5min":31.620999999999999,"15min":31.620999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":3.016,"5min":1.008,"15min":0.67800000000000005},"min":{"1min":0.38700000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":47.942999999999998,"5min":47.942999999999998,"15min":47.942999999999998},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon Dec 25 20:55:41 2023","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":0.78300000000000003,"15min":0.55400000000000005},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":31.036999999999999,"5min":31.036999999999999,"15min":31.036999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":2.181,"5min":0.80000000000000004,"15min":0.56799999999999995},"min":{"1min":0.42099999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":31.064,"5min":31.064,"15min":31.064},"last":0.499}]},{"osd":3,"last update":"Mon Dec 25 20:55:41 2023","interfaces":[{"interface":"back","average":{"1min":1.976,"5min":0.66800000000000004,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.249,"15min":0.249},"max":{"1min":30.975999999999999,"5min":30.975999999999999,"15min":30.975999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.6699999999999999,"5min":0.63,"15min":0.45300000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":24.087,"5min":24.087,"15min":24.087},"last":0.623}]},{"osd":4,"last update":"Mon Dec 25 20:55:41 2023","interfaces":[{"interface":"back","average":{"1min":1.7190000000000001,"5min":0.65100000000000002,"15min":0.47499999999999998},"min":{"1min":0.26400000000000001,"5min":0.253,"15min":0.253},"max":{"1min":25.238,"5min":25.238,"15min":25.238},"last":0.64800000000000002},{"interface":"front","average":{"1min":2.0739999999999998,"5min":0.75600000000000001,"15min":0.53300000000000003},"min":{"1min":0.32300000000000001,"5min":0.317,"15min":0.317},"max":{"1min":31.577000000000002,"5min":31.577000000000002,"15min":31.577000000000002},"last":0.60099999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640305,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:55:37 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48299999999999998,"15min":0.47499999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.498,"15min":0.48799999999999999},"min":{"1min":0.38200000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.47999999999999998}]},{"osd":1,"last update":"Mon Dec 25 20:55:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.51600000000000001,"15min":0.52200000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.88900000000000001,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47199999999999998,"15min":0.46200000000000002},"min":{"1min":0.33300000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Dec 25 20:55:37 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47399999999999998,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.65800000000000003,"5min":1.056,"15min":1.056},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.46200000000000002,"15min":0.44700000000000001},"min":{"1min":0.32400000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.39200000000000002}]},{"osd":3,"last update":"Mon Dec 25 20:55:37 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.375,"15min":0.38300000000000001},"min":{"1min":0.24199999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.72499999999999998,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.376,"5min":0.35799999999999998,"15min":0.34999999999999998},"min":{"1min":0.26200000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.33100000000000002}]},{"osd":5,"last update":"Mon Dec 25 20:55:37 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.46200000000000002,"15min":0.496},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60099999999999998,"5min":2.9500000000000002,"15min":2.9500000000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.48299999999999998,"15min":0.51700000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.597,"5min":3.0099999999999998,"15min":3.0099999999999998},"last":0.45700000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640305,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.46500000000000002,"15min":0.44700000000000001},"min":{"1min":0.309,"5min":0.245,"15min":0.245},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.45700000000000002,"15min":0.437},"min":{"1min":0.33000000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.45900000000000002}]},{"osd":1,"last update":"Mon Dec 25 20:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.45500000000000002,"15min":0.439},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.443,"15min":0.42899999999999999},"min":{"1min":0.315,"5min":0.31,"15min":0.31},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.46800000000000003}]},{"osd":2,"last update":"Mon Dec 25 20:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.45900000000000002,"15min":0.44500000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.45800000000000002,"15min":0.442},"min":{"1min":0.33000000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.502}]},{"osd":4,"last update":"Mon Dec 25 20:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.34300000000000003,"15min":0.33300000000000002},"min":{"1min":0.22800000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39000000000000001,"15min":0.38200000000000001},"min":{"1min":0.307,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.38700000000000001}]},{"osd":5,"last update":"Mon Dec 25 20:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26000000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.36199999999999999,"15min":0.35799999999999998},"min":{"1min":0.25800000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.43099999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640306,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.314,"15min":0.312},"min":{"1min":0.14099999999999999,"5min":0.14099999999999999,"15min":0.14099999999999999},"max":{"1min":0.41999999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.33800000000000002,"15min":0.33200000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.48399999999999999,"5min":0.495,"15min":0.495},"last":0.29199999999999998}]},{"osd":1,"last update":"Mon Dec 25 20:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.34999999999999998,"15min":0.35199999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.43099999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.33100000000000002,"15min":0.32700000000000001},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.38200000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.246}]},{"osd":3,"last update":"Mon Dec 25 20:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.46400000000000002,"15min":0.44400000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.70199999999999996,"5min":0.748,"15min":0.748},"last":0.434},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.46999999999999997,"15min":0.44900000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.52500000000000002}]},{"osd":4,"last update":"Mon Dec 25 20:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.47199999999999998,"15min":0.44800000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.76300000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.439},{"interface":"front","average":{"1min":0.498,"5min":0.48499999999999999,"15min":0.47299999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.66000000000000003,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.47599999999999998}]},{"osd":5,"last update":"Mon Dec 25 20:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.49399999999999999,"15min":0.48199999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65800000000000003,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.495,"15min":0.47699999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.73699999999999999,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.44800000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640305,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.309},"min":{"1min":0.223,"5min":0.186,"15min":0.186},"max":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"last":0.498},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.33700000000000002,"15min":0.33500000000000002},"min":{"1min":0.254,"5min":0.216,"15min":0.216},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.51800000000000002}]},{"osd":2,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.34300000000000003,"15min":0.34399999999999997},"min":{"1min":0.27000000000000002,"5min":0.216,"15min":0.216},"max":{"1min":0.41499999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.38},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35599999999999998,"15min":0.35199999999999998},"min":{"1min":0.249,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.45100000000000001,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.51900000000000002}]},{"osd":3,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.46800000000000003,"15min":0.44400000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.47699999999999998,"15min":0.46400000000000002},"min":{"1min":0.35399999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.64600000000000002,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.54500000000000004}]},{"osd":4,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51500000000000001,"15min":0.49299999999999999},"min":{"1min":0.376,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.51900000000000002,"15min":0.498},"min":{"1min":0.38200000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68300000000000005,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.57899999999999996}]},{"osd":5,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.49099999999999999,"15min":0.47899999999999998},"min":{"1min":0.38800000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.51100000000000001,"15min":0.49099999999999999},"min":{"1min":0.38400000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.53500000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640306,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:55:33 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.32700000000000001,"15min":0.32100000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33500000000000002,"15min":0.33200000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.32100000000000001}]},{"osd":2,"last update":"Mon Dec 25 20:55:39 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34899999999999998,"15min":0.34999999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.496,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.36199999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.52800000000000002,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.34599999999999997}]},{"osd":3,"last update":"Mon Dec 25 20:55:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.47599999999999998,"15min":0.46200000000000002},"min":{"1min":0.33800000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.61399999999999999,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.502,"15min":0.48899999999999999},"min":{"1min":0.39100000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.60599999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.41699999999999998}]},{"osd":4,"last update":"Mon Dec 25 20:55:33 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47499999999999998,"15min":0.46400000000000002},"min":{"1min":0.34999999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.60299999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46700000000000003,"15min":0.45700000000000002},"min":{"1min":0.36499999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.54400000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.375}]},{"osd":5,"last update":"Mon Dec 25 20:55:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.47699999999999998,"15min":0.46500000000000002},"min":{"1min":0.374,"5min":0.31,"15min":0.31},"max":{"1min":0.622,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.5,"15min":0.48999999999999999},"min":{"1min":0.38500000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.64500000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.435}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-25T20:56:27.733 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-25T20:52:33.281268+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=25, tm_hour=20, tm_min=52, tm_sec=33, tm_wday=0, tm_yday=359, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=25, tm_hour=20, tm_min=56, tm_sec=23, tm_wday=0, tm_yday=359, tm_isdst=0) 2023-12-25T20:56:27.734 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-25T20:52:33.281268+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=25, tm_hour=20, tm_min=52, tm_sec=33, tm_wday=0, tm_yday=359, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=25, tm_hour=20, tm_min=56, tm_sec=23, tm_wday=0, tm_yday=359, tm_isdst=0) 2023-12-25T20:56:27.734 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-25T20:52:33.281268+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=25, tm_hour=20, tm_min=52, tm_sec=33, tm_wday=0, tm_yday=359, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=25, tm_hour=20, tm_min=56, tm_sec=23, tm_wday=0, tm_yday=359, tm_isdst=0) 2023-12-25T20:56:27.734 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-25T20:52:33.281268+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=25, tm_hour=20, tm_min=52, tm_sec=33, tm_wday=0, tm_yday=359, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=25, tm_hour=20, tm_min=56, tm_sec=23, tm_wday=0, tm_yday=359, tm_isdst=0) 2023-12-25T20:56:27.734 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-25T20:52:32.273733+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=25, tm_hour=20, tm_min=52, tm_sec=32, tm_wday=0, tm_yday=359, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=25, tm_hour=20, tm_min=56, tm_sec=23, tm_wday=0, tm_yday=359, tm_isdst=0) 2023-12-25T20:56:27.734 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-25T20:52:33.281268+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=25, tm_hour=20, tm_min=52, tm_sec=33, tm_wday=0, tm_yday=359, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=25, tm_hour=20, tm_min=56, tm_sec=23, tm_wday=0, tm_yday=359, tm_isdst=0) 2023-12-25T20:56:27.734 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-25T20:52:33.281268+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=25, tm_hour=20, tm_min=52, tm_sec=33, tm_wday=0, tm_yday=359, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=25, tm_hour=20, tm_min=56, tm_sec=23, tm_wday=0, tm_yday=359, tm_isdst=0) 2023-12-25T20:56:27.735 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-25T20:52:33.281268+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=25, tm_hour=20, tm_min=52, tm_sec=33, tm_wday=0, tm_yday=359, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=25, tm_hour=20, tm_min=56, tm_sec=23, tm_wday=0, tm_yday=359, tm_isdst=0) 2023-12-25T20:56:27.735 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-25T20:52:33.281268+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=25, tm_hour=20, tm_min=52, tm_sec=33, tm_wday=0, tm_yday=359, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=25, tm_hour=20, tm_min=56, tm_sec=23, tm_wday=0, tm_yday=359, tm_isdst=0) 2023-12-25T20:56:27.735 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-25T20:56:47.736 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-25T20:56:48.041 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-12-25T20:56:48.043 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-12-25T20:56:48.054 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2023-12-25T20:56:47.790775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786992,"kb_used_data":4416,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564444048,"statfs":{"total":579820584960,"available":577990705152,"internally_reserved":0,"allocated":4521984,"data_stored":1943586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":57,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-25T20:56:28.326954+0000","last_change":"2023-12-25T20:56:28.326954+0000","last_active":"2023-12-25T20:56:28.326954+0000","last_peered":"2023-12-25T20:56:28.326954+0000","last_clean":"2023-12-25T20:56:28.326954+0000","last_became_active":"2023-12-25T20:52:34.300369+0000","last_became_peered":"2023-12-25T20:52:34.300369+0000","last_unstale":"2023-12-25T20:56:28.326954+0000","last_undegraded":"2023-12-25T20:56:28.326954+0000","last_fullsized":"2023-12-25T20:56:28.326954+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:56:28.326834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:56:28.326834+0000","last_clean_scrub_stamp":"2023-12-25T20:56:28.326834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":61,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-25T20:56:25.491032+0000","last_change":"2023-12-25T20:56:25.491032+0000","last_active":"2023-12-25T20:56:25.491032+0000","last_peered":"2023-12-25T20:56:25.491032+0000","last_clean":"2023-12-25T20:56:25.491032+0000","last_became_active":"2023-12-25T20:52:34.295559+0000","last_became_peered":"2023-12-25T20:52:34.295559+0000","last_unstale":"2023-12-25T20:56:25.491032+0000","last_undegraded":"2023-12-25T20:56:25.491032+0000","last_fullsized":"2023-12-25T20:56:25.491032+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:56:25.490953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:56:25.490953+0000","last_clean_scrub_stamp":"2023-12-25T20:56:25.490953+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":58,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-25T20:56:28.012850+0000","last_change":"2023-12-25T20:56:28.012850+0000","last_active":"2023-12-25T20:56:28.012850+0000","last_peered":"2023-12-25T20:56:28.012850+0000","last_clean":"2023-12-25T20:56:28.012850+0000","last_became_active":"2023-12-25T20:52:34.300344+0000","last_became_peered":"2023-12-25T20:52:34.300344+0000","last_unstale":"2023-12-25T20:56:28.012850+0000","last_undegraded":"2023-12-25T20:56:28.012850+0000","last_fullsized":"2023-12-25T20:56:28.012850+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:56:28.012754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:56:28.012754+0000","last_clean_scrub_stamp":"2023-12-25T20:56:28.012754+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":61,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-25T20:56:34.454585+0000","last_change":"2023-12-25T20:56:34.454585+0000","last_active":"2023-12-25T20:56:34.454585+0000","last_peered":"2023-12-25T20:56:34.454585+0000","last_clean":"2023-12-25T20:56:34.454585+0000","last_became_active":"2023-12-25T20:52:34.297376+0000","last_became_peered":"2023-12-25T20:52:34.297376+0000","last_unstale":"2023-12-25T20:56:34.454585+0000","last_undegraded":"2023-12-25T20:56:34.454585+0000","last_fullsized":"2023-12-25T20:56:34.454585+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:56:34.454487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:56:34.454487+0000","last_clean_scrub_stamp":"2023-12-25T20:56:34.454487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-25T20:56:28.521121+0000","last_change":"2023-12-25T20:56:28.521121+0000","last_active":"2023-12-25T20:56:28.521121+0000","last_peered":"2023-12-25T20:56:28.521121+0000","last_clean":"2023-12-25T20:56:28.521121+0000","last_became_active":"2023-12-25T20:52:33.421006+0000","last_became_peered":"2023-12-25T20:52:33.421006+0000","last_unstale":"2023-12-25T20:56:28.521121+0000","last_undegraded":"2023-12-25T20:56:28.521121+0000","last_fullsized":"2023-12-25T20:56:28.521121+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:56:28.521011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:56:28.521011+0000","last_clean_scrub_stamp":"2023-12-25T20:56:28.521011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":56,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-25T20:56:30.289389+0000","last_change":"2023-12-25T20:56:30.289389+0000","last_active":"2023-12-25T20:56:30.289389+0000","last_peered":"2023-12-25T20:56:30.289389+0000","last_clean":"2023-12-25T20:56:30.289389+0000","last_became_active":"2023-12-25T20:52:34.294123+0000","last_became_peered":"2023-12-25T20:52:34.294123+0000","last_unstale":"2023-12-25T20:56:30.289389+0000","last_undegraded":"2023-12-25T20:56:30.289389+0000","last_fullsized":"2023-12-25T20:56:30.289389+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:56:30.289293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:56:30.289293+0000","last_clean_scrub_stamp":"2023-12-25T20:56:30.289293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":61,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-25T20:56:35.496224+0000","last_change":"2023-12-25T20:56:35.496224+0000","last_active":"2023-12-25T20:56:35.496224+0000","last_peered":"2023-12-25T20:56:35.496224+0000","last_clean":"2023-12-25T20:56:35.496224+0000","last_became_active":"2023-12-25T20:52:34.300607+0000","last_became_peered":"2023-12-25T20:52:34.300607+0000","last_unstale":"2023-12-25T20:56:35.496224+0000","last_undegraded":"2023-12-25T20:56:35.496224+0000","last_fullsized":"2023-12-25T20:56:35.496224+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:56:35.496104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:56:35.496104+0000","last_clean_scrub_stamp":"2023-12-25T20:56:35.496104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":58,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-25T20:56:29.152491+0000","last_change":"2023-12-25T20:56:29.152491+0000","last_active":"2023-12-25T20:56:29.152491+0000","last_peered":"2023-12-25T20:56:29.152491+0000","last_clean":"2023-12-25T20:56:29.152491+0000","last_became_active":"2023-12-25T20:52:34.293804+0000","last_became_peered":"2023-12-25T20:52:34.293804+0000","last_unstale":"2023-12-25T20:56:29.152491+0000","last_undegraded":"2023-12-25T20:56:29.152491+0000","last_fullsized":"2023-12-25T20:56:29.152491+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:56:29.152390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:56:29.152390+0000","last_clean_scrub_stamp":"2023-12-25T20:56:29.152390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":61,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-25T20:56:37.505472+0000","last_change":"2023-12-25T20:56:37.505472+0000","last_active":"2023-12-25T20:56:37.505472+0000","last_peered":"2023-12-25T20:56:37.505472+0000","last_clean":"2023-12-25T20:56:37.505472+0000","last_became_active":"2023-12-25T20:52:34.297388+0000","last_became_peered":"2023-12-25T20:52:34.297388+0000","last_unstale":"2023-12-25T20:56:37.505472+0000","last_undegraded":"2023-12-25T20:56:37.505472+0000","last_fullsized":"2023-12-25T20:56:37.505472+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-25T20:56:37.505382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-25T20:56:37.505382+0000","last_clean_scrub_stamp":"2023-12-25T20:56:37.505382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640310,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:56:43 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.753,"15min":0.56200000000000006},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77400000000000002,"5min":22.896999999999998,"15min":22.896999999999998},"last":0.443},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.79500000000000004,"15min":0.59099999999999997},"min":{"1min":0.35299999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.67700000000000005,"5min":25.178000000000001,"15min":25.178000000000001},"last":0.40400000000000003}]},{"osd":1,"last update":"Mon Dec 25 20:56:43 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.82499999999999996,"15min":0.59299999999999997},"min":{"1min":0.34200000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.61299999999999999,"5min":31.620999999999999,"15min":31.620999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":1.012,"15min":0.67900000000000005},"min":{"1min":0.41899999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.61099999999999999,"5min":47.942999999999998,"15min":47.942999999999998},"last":0.42799999999999999}]},{"osd":2,"last update":"Mon Dec 25 20:56:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.80100000000000005,"15min":0.56000000000000005},"min":{"1min":0.41199999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.65800000000000003,"5min":31.036999999999999,"15min":31.036999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.82199999999999995,"15min":0.57499999999999996},"min":{"1min":0.39900000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68899999999999995,"5min":31.064,"15min":31.064},"last":0.50900000000000001}]},{"osd":3,"last update":"Mon Dec 25 20:56:43 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.68500000000000005,"15min":0.45300000000000001},"min":{"1min":0.32500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.61899999999999999,"5min":30.975999999999999,"15min":30.975999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.65000000000000002,"15min":0.46000000000000002},"min":{"1min":0.315,"5min":0.25,"15min":0.25},"max":{"1min":0.64800000000000002,"5min":24.087,"15min":24.087},"last":0.36599999999999999}]},{"osd":4,"last update":"Mon Dec 25 20:56:43 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.65800000000000003,"15min":0.47699999999999998},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.64800000000000002,"5min":25.238,"15min":25.238},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.76700000000000002,"15min":0.53700000000000003},"min":{"1min":0.376,"5min":0.317,"15min":0.317},"max":{"1min":0.60999999999999999,"5min":31.577000000000002,"15min":31.577000000000002},"last":0.45600000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640310,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48699999999999999,"15min":0.47699999999999998},"min":{"1min":0.33300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.66200000000000003,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.50700000000000001,"15min":0.49099999999999999},"min":{"1min":0.38300000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.63500000000000001,"5min":1.579,"15min":1.579},"last":0.67300000000000004}]},{"osd":1,"last update":"Mon Dec 25 20:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51200000000000001,"15min":0.52000000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.67600000000000005,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.48299999999999998,"15min":0.46500000000000002},"min":{"1min":0.35599999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.66400000000000003,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.68300000000000005}]},{"osd":2,"last update":"Mon Dec 25 20:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47799999999999998},"min":{"1min":0.32700000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.59699999999999998,"5min":1.056,"15min":1.056},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46500000000000002,"15min":0.44800000000000001},"min":{"1min":0.33600000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.59799999999999998,"5min":1.538,"15min":1.538},"last":0.60699999999999998}]},{"osd":3,"last update":"Mon Dec 25 20:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.372,"15min":0.38200000000000001},"min":{"1min":0.26300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.57199999999999995,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.36799999999999999,"15min":0.35299999999999998},"min":{"1min":0.27500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68700000000000006,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.35299999999999998}]},{"osd":5,"last update":"Mon Dec 25 20:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.44600000000000001,"15min":0.48999999999999999},"min":{"1min":0.33100000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.65700000000000003,"5min":2.9500000000000002,"15min":2.9500000000000002},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46800000000000003,"15min":0.51200000000000001},"min":{"1min":0.27700000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.71299999999999997,"5min":3.0099999999999998,"15min":3.0099999999999998},"last":0.40899999999999997}]}]},{"osd":3,"up_from":11,"seq":47244640310,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47499999999999998,"15min":0.45100000000000001},"min":{"1min":0.39500000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.63200000000000001,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.47099999999999997,"15min":0.441},"min":{"1min":0.38300000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72299999999999998,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.55100000000000005}]},{"osd":1,"last update":"Mon Dec 25 20:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.46800000000000003,"15min":0.443},"min":{"1min":0.375,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.622,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.45500000000000002,"15min":0.433},"min":{"1min":0.38400000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.60799999999999998,"5min":0.876,"15min":0.876},"last":0.57299999999999995}]},{"osd":2,"last update":"Mon Dec 25 20:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.46999999999999997,"15min":0.44800000000000001},"min":{"1min":0.374,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.622,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.495,"5min":0.46999999999999997,"15min":0.44600000000000001},"min":{"1min":0.39500000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70799999999999996,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.628}]},{"osd":4,"last update":"Mon Dec 25 20:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.35799999999999998,"15min":0.33800000000000002},"min":{"1min":0.312,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.54800000000000004,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.438},{"interface":"front","average":{"1min":0.439,"5min":0.40200000000000002,"15min":0.38600000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64800000000000002,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.47099999999999997}]},{"osd":5,"last update":"Mon Dec 25 20:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.36799999999999999,"15min":0.36299999999999999},"min":{"1min":0.26000000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.56599999999999995,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.375,"15min":0.36199999999999999},"min":{"1min":0.28499999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.63700000000000001,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.38300000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640311,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:56:43 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.31900000000000001,"15min":0.314},"min":{"1min":0.21199999999999999,"5min":0.14099999999999999,"15min":0.14099999999999999},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.34999999999999998,"15min":0.33600000000000002},"min":{"1min":0.27200000000000002,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.52300000000000002}]},{"osd":1,"last update":"Mon Dec 25 20:56:43 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34799999999999998,"15min":0.35099999999999998},"min":{"1min":0.23799999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.51000000000000001,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.33300000000000002,"15min":0.32800000000000001},"min":{"1min":0.23300000000000001,"5min":0.184,"15min":0.184},"max":{"1min":0.48399999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.441}]},{"osd":3,"last update":"Mon Dec 25 20:56:43 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.47999999999999998,"15min":0.44900000000000001},"min":{"1min":0.39000000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.68400000000000005,"5min":0.748,"15min":0.748},"last":0.5},{"interface":"front","average":{"1min":0.505,"5min":0.48399999999999999,"15min":0.45400000000000001},"min":{"1min":0.35999999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.68400000000000005,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.57499999999999996}]},{"osd":4,"last update":"Mon Dec 25 20:56:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.49299999999999999,"15min":0.45500000000000002},"min":{"1min":0.39500000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.76400000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.496,"15min":0.47699999999999998},"min":{"1min":0.38900000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.69499999999999995,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.53600000000000003}]},{"osd":5,"last update":"Mon Dec 25 20:56:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.50800000000000001,"15min":0.48699999999999999},"min":{"1min":0.36199999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.50800000000000001,"15min":0.48099999999999998},"min":{"1min":0.36699999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.72999999999999998,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.51300000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640310,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Dec 25 20:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.32200000000000001,"15min":0.312},"min":{"1min":0.22500000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.34100000000000003,"15min":0.33700000000000002},"min":{"1min":0.26600000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.51800000000000002,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.38900000000000001}]},{"osd":2,"last update":"Mon Dec 25 20:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.25900000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.47399999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.372},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.36299999999999999,"15min":0.35399999999999998},"min":{"1min":0.251,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.51900000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.34699999999999998}]},{"osd":3,"last update":"Mon Dec 25 20:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.47899999999999998,"15min":0.44800000000000001},"min":{"1min":0.33100000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.65200000000000002,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.437},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.47999999999999998,"15min":0.46500000000000002},"min":{"1min":0.35099999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.55400000000000005,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.42199999999999999}]},{"osd":4,"last update":"Mon Dec 25 20:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52600000000000002,"15min":0.497},"min":{"1min":0.38,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72899999999999998,"5min":0.747,"15min":0.747},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53000000000000003,"15min":0.502},"min":{"1min":0.39300000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.71399999999999997,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.45600000000000002}]},{"osd":5,"last update":"Mon Dec 25 20:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.502,"15min":0.48199999999999998},"min":{"1min":0.38200000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52400000000000002,"15min":0.495},"min":{"1min":0.45600000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.68899999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.51000000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640311,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 25 20:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32900000000000001,"15min":0.32100000000000001},"min":{"1min":0.23200000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.47299999999999998,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.33500000000000002,"15min":0.33200000000000002},"min":{"1min":0.251,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.50900000000000001,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.35999999999999999}]},{"osd":2,"last update":"Mon Dec 25 20:56:41 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.34699999999999998,"15min":0.34899999999999998},"min":{"1min":0.27500000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.46100000000000002,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.26700000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.52900000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.374}]},{"osd":3,"last update":"Mon Dec 25 20:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.496,"15min":0.46899999999999997},"min":{"1min":0.34000000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.77600000000000002,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.52000000000000002,"15min":0.495},"min":{"1min":0.41599999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78000000000000003,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.52400000000000002}]},{"osd":4,"last update":"Mon Dec 25 20:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.48599999999999999,"15min":0.46800000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.69399999999999995,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.48099999999999998,"15min":0.46200000000000002},"min":{"1min":0.36499999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70999999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.50600000000000001}]},{"osd":5,"last update":"Mon Dec 25 20:56:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.498,"15min":0.47199999999999998},"min":{"1min":0.38200000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.76700000000000002,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.52200000000000002,"15min":0.498},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.56699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-25T20:56:48.056 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-12-25T20:56:48.365 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-25T20:56:48.366 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-25T20:56:48.366 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-25T20:56:48.366 INFO:teuthology.orchestra.run:waiting for 300 2023-12-25T20:56:48.398 INFO:tasks.ceph.osd.0:Stopped 2023-12-25T20:56:48.399 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-25T20:56:48.399 INFO:teuthology.orchestra.run:waiting for 300 2023-12-25T20:56:48.461 INFO:tasks.ceph.osd.1:Stopped 2023-12-25T20:56:48.461 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-25T20:56:48.462 INFO:teuthology.orchestra.run:waiting for 300 2023-12-25T20:56:48.526 INFO:tasks.ceph.osd.2:Stopped 2023-12-25T20:56:48.526 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-12-25T20:56:48.526 INFO:teuthology.orchestra.run:waiting for 300 2023-12-25T20:56:48.556 INFO:tasks.ceph.osd.3:Stopped 2023-12-25T20:56:48.556 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-12-25T20:56:48.557 INFO:teuthology.orchestra.run:waiting for 300 2023-12-25T20:56:48.628 INFO:tasks.ceph.osd.4:Stopped 2023-12-25T20:56:48.629 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-12-25T20:56:48.629 INFO:teuthology.orchestra.run:waiting for 300 2023-12-25T20:56:48.693 INFO:tasks.ceph.osd.5:Stopped 2023-12-25T20:56:48.694 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-25T20:56:48.694 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-25T20:56:48.694 INFO:teuthology.orchestra.run:waiting for 300 2023-12-25T20:56:48.757 INFO:tasks.ceph.mgr.x:Stopped 2023-12-25T20:56:48.758 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-25T20:56:48.758 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-25T20:56:48.758 INFO:teuthology.orchestra.run:waiting for 300 2023-12-25T20:56:48.768 INFO:tasks.ceph.mon.a:Stopped 2023-12-25T20:56:48.768 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-25T20:56:48.768 INFO:teuthology.orchestra.run:waiting for 300 2023-12-25T20:56:48.818 INFO:tasks.ceph.mon.c:Stopped 2023-12-25T20:56:48.819 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-25T20:56:48.819 INFO:teuthology.orchestra.run:waiting for 300 2023-12-25T20:56:48.831 INFO:tasks.ceph.mon.b:Stopped 2023-12-25T20:56:48.831 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-25T20:56:48.831 DEBUG:teuthology.orchestra.run.smithi153:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(AUTH_BAD_CAPS\)' | head -n 1 2023-12-25T20:56:48.890 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi153.front.sepia.ceph.com 2023-12-25T20:56:48.891 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-25T20:56:49.087 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi153.front.sepia.ceph.com 2023-12-25T20:56:49.087 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-25T20:56:49.203 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi153.front.sepia.ceph.com 2023-12-25T20:56:49.204 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-25T20:56:49.263 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi191.front.sepia.ceph.com 2023-12-25T20:56:49.263 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-12-25T20:56:49.443 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi191.front.sepia.ceph.com 2023-12-25T20:56:49.444 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-12-25T20:56:49.520 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi191.front.sepia.ceph.com 2023-12-25T20:56:49.520 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-12-25T20:56:49.572 INFO:tasks.ceph:Archiving mon data... 2023-12-25T20:56:49.573 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500952/data/mon.a.tgz 2023-12-25T20:56:49.573 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2023-12-25T20:56:49.592 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.Y43SuQn0CU 2023-12-25T20:56:49.592 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Y43SuQn0CU 2023-12-25T20:56:49.788 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.Y43SuQn0CU 2023-12-25T20:56:49.877 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.Y43SuQn0CU is 535KB 2023-12-25T20:56:49.976 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.Y43SuQn0CU 2023-12-25T20:56:49.993 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500952/data/mon.c.tgz 2023-12-25T20:56:49.994 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2023-12-25T20:56:50.050 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.79HkFgnvKT 2023-12-25T20:56:50.051 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.79HkFgnvKT 2023-12-25T20:56:50.218 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.79HkFgnvKT 2023-12-25T20:56:50.306 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.79HkFgnvKT is 535KB 2023-12-25T20:56:50.406 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.79HkFgnvKT 2023-12-25T20:56:50.424 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500952/data/mon.b.tgz 2023-12-25T20:56:50.424 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2023-12-25T20:56:50.446 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.pMGiU6MBny 2023-12-25T20:56:50.447 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.pMGiU6MBny 2023-12-25T20:56:50.636 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.pMGiU6MBny 2023-12-25T20:56:50.724 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.pMGiU6MBny is 535KB 2023-12-25T20:56:50.796 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.pMGiU6MBny 2023-12-25T20:56:50.814 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-25T20:56:50.815 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-25T20:56:50.817 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-25T20:56:50.879 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-12-25T20:56:50.880 DEBUG:teuthology.orchestra.run.smithi153:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-12-25T20:56:50.883 DEBUG:teuthology.orchestra.run.smithi191:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-12-25T20:56:50.909 INFO:teuthology.orchestra.run.smithi153.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-12-25T20:56:50.947 INFO:teuthology.orchestra.run.smithi191.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-12-25T20:56:50.952 INFO:tasks.ceph:Archiving crash dumps... 2023-12-25T20:56:50.954 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500952/remote/smithi153/crash 2023-12-25T20:56:50.955 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-25T20:56:50.982 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500952/remote/smithi191/crash 2023-12-25T20:56:50.983 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-25T20:56:51.018 INFO:tasks.ceph:Compressing logs... 2023-12-25T20:56:51.019 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-25T20:56:51.027 DEBUG:teuthology.orchestra.run.smithi191:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-25T20:56:51.079 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121571.log 2023-12-25T20:56:51.080 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119622.log 2023-12-25T20:56:51.080 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.121571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120983.log 2023-12-25T20:56:51.080 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121571.log.gz 2023-12-25T20:56:51.080 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.119622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122306.log 2023-12-25T20:56:51.080 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119622.log.gz 2023-12-25T20:56:51.080 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.120983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121522.log 2023-12-25T20:56:51.081 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120983.log.gz 2023-12-25T20:56:51.081 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.122306.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122306.log.gz 2023-12-25T20:56:51.081 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119000.log 2023-12-25T20:56:51.081 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.121522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117136.log 2023-12-25T20:56:51.081 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121522.log.gz 2023-12-25T20:56:51.081 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.119000.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119000.log.gzgzip 2023-12-25T20:56:51.081 INFO:teuthology.orchestra.run.smithi153.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.119867.log 2023-12-25T20:56:51.081 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.117136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117822.log 2023-12-25T20:56:51.082 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117136.log.gz 2023-12-25T20:56:51.082 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.119867.log: gzip -5 --verbose 0.0%-- -- replaced with /var/log/ceph/ceph-client.admin.119867.log.gz/var/log/ceph/ceph-client.admin.117038.log 2023-12-25T20:56:51.082 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-12-25T20:56:51.082 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.117822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120483.log 2023-12-25T20:56:51.082 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117822.log.gz 2023-12-25T20:56:51.082 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.117038.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117038.log.gz 2023-12-25T20:56:51.082 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120434.log 2023-12-25T20:56:51.082 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.120483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122208.log 2023-12-25T20:56:51.082 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120483.log.gz 2023-12-25T20:56:51.083 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.120434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118853.log 2023-12-25T20:56:51.083 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120434.log.gz 2023-12-25T20:56:51.083 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.122208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118804.log 2023-12-25T20:56:51.083 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122208.log.gz 2023-12-25T20:56:51.083 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.118853.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118853.log.gz 2023-12-25T20:56:51.083 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118165.log 2023-12-25T20:56:51.083 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.118804.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-25T20:56:51.083 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118804.log.gz 2023-12-25T20:56:51.083 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.118165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118165.log.gzgzip 2023-12-25T20:56:51.084 INFO:teuthology.orchestra.run.smithi153.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.117528.log 2023-12-25T20:56:51.084 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117087.log 2023-12-25T20:56:51.084 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.117528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117528.log.gz 2023-12-25T20:56:51.084 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118459.log 2023-12-25T20:56:51.084 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.117087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114477.log 2023-12-25T20:56:51.084 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117087.log.gz 2023-12-25T20:56:51.084 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.118459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118459.log.gz 2023-12-25T20:56:51.084 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118951.log 2023-12-25T20:56:51.084 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.114477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118067.log 2023-12-25T20:56:51.084 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114477.log.gz 2023-12-25T20:56:51.085 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.118951.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.119400.log 2023-12-25T20:56:51.085 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.118951.log.gz 2023-12-25T20:56:51.085 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.118067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122510.log 2023-12-25T20:56:51.085 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118067.log.gz 2023-12-25T20:56:51.085 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.119400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120294.log 2023-12-25T20:56:51.085 INFO:teuthology.orchestra.run.smithi153.stderr: 27.0% -- replaced with /var/log/ceph/ceph-client.admin.119400.log.gz 2023-12-25T20:56:51.085 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.122510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118410.log 2023-12-25T20:56:51.085 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122510.log.gz 2023-12-25T20:56:51.086 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.120294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118018.log 2023-12-25T20:56:51.086 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120294.log.gz 2023-12-25T20:56:51.086 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.118410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118410.log.gz 2023-12-25T20:56:51.086 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118116.log 2023-12-25T20:56:51.086 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.118018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121767.log 2023-12-25T20:56:51.086 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118018.log.gz 2023-12-25T20:56:51.086 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.118116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120384.log 2023-12-25T20:56:51.087 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118116.log.gz 2023-12-25T20:56:51.087 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.121767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118214.log 2023-12-25T20:56:51.087 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121767.log.gz 2023-12-25T20:56:51.087 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.120384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117871.log 2023-12-25T20:56:51.087 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120384.log.gz 2023-12-25T20:56:51.087 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.118214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118214.log.gz 2023-12-25T20:56:51.087 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122257.log 2023-12-25T20:56:51.087 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.117871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118704.log 2023-12-25T20:56:51.087 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117871.log.gz 2023-12-25T20:56:51.088 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.122257.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-25T20:56:51.088 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122257.log.gz 2023-12-25T20:56:51.088 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.118704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119573.log 2023-12-25T20:56:51.088 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118704.log.gz 2023-12-25T20:56:51.088 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119515.log 2023-12-25T20:56:51.088 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.119573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118508.log 2023-12-25T20:56:51.088 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119573.log.gz 2023-12-25T20:56:51.088 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.119515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122453.log 2023-12-25T20:56:51.088 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119515.log.gz 2023-12-25T20:56:51.089 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.118508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118606.log 2023-12-25T20:56:51.089 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118508.log.gz 2023-12-25T20:56:51.089 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.122453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117577.log 2023-12-25T20:56:51.089 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122453.log.gz 2023-12-25T20:56:51.089 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.118606.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118606.log.gz 2023-12-25T20:56:51.089 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121228.log 2023-12-25T20:56:51.090 INFO:teuthology.orchestra.run.smithi153.stderr: 93.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-25T20:56:51.090 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.117577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117626.log 2023-12-25T20:56:51.090 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117577.log.gz 2023-12-25T20:56:51.090 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.121228.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.121228.log.gz/var/log/ceph/ceph-osd.0.log 2023-12-25T20:56:51.090 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-12-25T20:56:51.090 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.117626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118902.log 2023-12-25T20:56:51.090 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117626.log.gz 2023-12-25T20:56:51.090 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114117.log 2023-12-25T20:56:51.090 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.118902.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118902.log.gz 2023-12-25T20:56:51.091 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119098.log 2023-12-25T20:56:51.091 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.114117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121718.log 2023-12-25T20:56:51.091 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114117.log.gz 2023-12-25T20:56:51.091 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.119098.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.119098.log.gz 2023-12-25T20:56:51.091 INFO:teuthology.orchestra.run.smithi153.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.122355.log 2023-12-25T20:56:51.091 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.121718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120162.log 2023-12-25T20:56:51.091 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121718.log.gz 2023-12-25T20:56:51.091 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.122355.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122355.log.gz 2023-12-25T20:56:51.091 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117332.log 2023-12-25T20:56:51.092 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.120162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119818.log 2023-12-25T20:56:51.092 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120162.log.gz 2023-12-25T20:56:51.092 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.117332.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.117332.log.gz-- 2023-12-25T20:56:51.092 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.119147.log 2023-12-25T20:56:51.092 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.119818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121081.log 2023-12-25T20:56:51.092 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119818.log.gz 2023-12-25T20:56:51.092 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.119147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117234.log 2023-12-25T20:56:51.092 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119147.log.gz 2023-12-25T20:56:51.093 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.121081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121326.log 2023-12-25T20:56:51.093 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121081.log.gz 2023-12-25T20:56:51.093 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.117234.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.117234.log.gz-5 2023-12-25T20:56:51.093 INFO:teuthology.orchestra.run.smithi153.stderr:--verbose -- /var/log/ceph/ceph-client.admin.120014.log 2023-12-25T20:56:51.093 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.121326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119769.log 2023-12-25T20:56:51.093 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121326.log.gz 2023-12-25T20:56:51.093 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.120014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119720.log 2023-12-25T20:56:51.093 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120014.log.gz 2023-12-25T20:56:51.093 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.119769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119245.log 2023-12-25T20:56:51.094 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119769.log.gz 2023-12-25T20:56:51.094 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.119720.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119720.log.gz 2023-12-25T20:56:51.094 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121375.log 2023-12-25T20:56:51.094 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.119245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117724.log 2023-12-25T20:56:51.094 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119245.log.gz 2023-12-25T20:56:51.094 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119916.log 2023-12-25T20:56:51.094 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.121375.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121375.log.gz 2023-12-25T20:56:51.094 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.117724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121914.log 2023-12-25T20:56:51.095 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117724.log.gz 2023-12-25T20:56:51.095 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121130.log 2023-12-25T20:56:51.095 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.119916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119916.log.gz 2023-12-25T20:56:51.095 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117920.log 2023-12-25T20:56:51.095 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.121914.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121914.log.gz 2023-12-25T20:56:51.095 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.121130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118557.log 2023-12-25T20:56:51.095 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121130.log.gz 2023-12-25T20:56:51.095 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.117920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117283.log 2023-12-25T20:56:51.096 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117920.log.gz 2023-12-25T20:56:51.096 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121424.log 2023-12-25T20:56:51.096 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.118557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118557.log.gz 2023-12-25T20:56:51.096 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.117283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117430.log 2023-12-25T20:56:51.096 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117283.log.gz 2023-12-25T20:56:51.096 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.121424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120885.log 2023-12-25T20:56:51.096 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121424.log.gz 2023-12-25T20:56:51.096 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.117430.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.117430.log.gz/var/log/ceph/ceph-client.admin.120632.log 2023-12-25T20:56:51.096 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-12-25T20:56:51.097 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.120885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118361.log 2023-12-25T20:56:51.097 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120885.log.gz 2023-12-25T20:56:51.097 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.120632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116969.log 2023-12-25T20:56:51.097 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120632.log.gz 2023-12-25T20:56:51.097 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.118361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117675.log 2023-12-25T20:56:51.097 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118361.log.gz 2023-12-25T20:56:51.097 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.116969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121669.log 2023-12-25T20:56:51.097 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116969.log.gz 2023-12-25T20:56:51.098 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.117675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121277.log 2023-12-25T20:56:51.098 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117675.log.gz 2023-12-25T20:56:51.098 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.121669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118263.log 2023-12-25T20:56:51.098 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121669.log.gz 2023-12-25T20:56:51.098 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.121277.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.109414.log 2023-12-25T20:56:51.098 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121277.log.gz 2023-12-25T20:56:51.098 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.118263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114393.log 2023-12-25T20:56:51.098 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118263.log.gz 2023-12-25T20:56:51.098 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.tmp-client.admin.109414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120738.log 2023-12-25T20:56:51.099 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.109414.log.gz 2023-12-25T20:56:51.099 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.114393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117773.log 2023-12-25T20:56:51.099 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114393.log.gz 2023-12-25T20:56:51.099 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.120738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120532.log 2023-12-25T20:56:51.099 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120738.log.gz 2023-12-25T20:56:51.099 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.117773.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-12-25T20:56:51.099 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117773.log.gz 2023-12-25T20:56:51.099 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.120532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117479.log 2023-12-25T20:56:51.099 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120532.log.gz 2023-12-25T20:56:51.100 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120063.log 2023-12-25T20:56:51.100 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.117479.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117479.log.gz 2023-12-25T20:56:51.100 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118655.log 2023-12-25T20:56:51.100 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121816.log 2023-12-25T20:56:51.100 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.120063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120063.log.gz 2023-12-25T20:56:51.100 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.118655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117185.log 2023-12-25T20:56:51.100 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118655.log.gz 2023-12-25T20:56:51.100 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.121816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122404.log 2023-12-25T20:56:51.100 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121816.log.gz 2023-12-25T20:56:51.101 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.117185.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.118753.log 2023-12-25T20:56:51.101 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.117185.log.gz 2023-12-25T20:56:51.101 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.122404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122061.log 2023-12-25T20:56:51.101 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122404.log.gz 2023-12-25T20:56:51.101 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.118753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118753.log.gz 2023-12-25T20:56:51.101 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-12-25T20:56:51.102 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.122061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122557.log 2023-12-25T20:56:51.102 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122061.log.gz 2023-12-25T20:56:51.102 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119302.log 2023-12-25T20:56:51.102 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.122557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121179.log 2023-12-25T20:56:51.102 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122557.log.gz 2023-12-25T20:56:51.102 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.119302.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119302.log.gz 2023-12-25T20:56:51.102 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.121179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121179.log.gz 2023-12-25T20:56:51.102 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121620.log 2023-12-25T20:56:51.103 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120787.log 2023-12-25T20:56:51.103 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.121620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121620.log.gz 2023-12-25T20:56:51.103 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.120787.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120787.log.gz 2023-12-25T20:56:51.103 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-25T20:56:51.103 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120934.log 2023-12-25T20:56:51.111 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117969.log 2023-12-25T20:56:51.111 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.120934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120934.log.gz 2023-12-25T20:56:51.112 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.117969.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117969.log.gz 2023-12-25T20:56:51.112 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119196.log 2023-12-25T20:56:51.112 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119049.log 2023-12-25T20:56:51.112 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.119196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119196.log.gz 2023-12-25T20:56:51.112 INFO:teuthology.orchestra.run.smithi153.stderr: 91.9% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-25T20:56:51.112 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.119049.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119049.log.gz 2023-12-25T20:56:51.112 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121032.log 2023-12-25T20:56:51.112 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120836.log 2023-12-25T20:56:51.113 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.121032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121865.log 2023-12-25T20:56:51.113 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121032.log.gz 2023-12-25T20:56:51.113 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.120836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120112.log 2023-12-25T20:56:51.113 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120836.log.gz 2023-12-25T20:56:51.113 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.121865.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121865.log.gz 2023-12-25T20:56:51.113 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.120112.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120112.log.gz 2023-12-25T20:56:51.113 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119671.log 2023-12-25T20:56:51.113 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120212.log 2023-12-25T20:56:51.114 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.119671.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119671.log.gz 2023-12-25T20:56:51.114 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.120212.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120212.log.gz 2023-12-25T20:56:51.114 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122110.log 2023-12-25T20:56:51.114 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120681.log 2023-12-25T20:56:51.114 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.122110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119965.log 2023-12-25T20:56:51.114 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122110.log.gz 2023-12-25T20:56:51.114 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.120681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118312.log 2023-12-25T20:56:51.114 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120681.log.gz 2023-12-25T20:56:51.114 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.119965.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119965.log.gz 2023-12-25T20:56:51.115 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.118312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118312.log.gz 2023-12-25T20:56:51.115 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117381.log 2023-12-25T20:56:51.115 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-25T20:56:51.115 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.117381.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117381.log.gz 2023-12-25T20:56:51.115 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121963.log 2023-12-25T20:56:51.115 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122012.log 2023-12-25T20:56:51.115 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.121963.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121963.log.gz 2023-12-25T20:56:51.115 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.122012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122012.log.gz 2023-12-25T20:56:51.116 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121473.log 2023-12-25T20:56:51.116 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122159.log 2023-12-25T20:56:51.116 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.121473.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121473.log.gz 2023-12-25T20:56:51.116 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.122159.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122159.log.gz 2023-12-25T20:56:51.116 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119351.log 2023-12-25T20:56:51.116 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120581.log 2023-12-25T20:56:51.116 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.119351.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119351.log.gz 2023-12-25T20:56:51.116 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.120581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120581.log.gz 2023-12-25T20:56:51.134 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118809.log 2023-12-25T20:56:51.135 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117607.log 2023-12-25T20:56:51.135 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117513.log 2023-12-25T20:56:51.135 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118809.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118809.log.gz 2023-12-25T20:56:51.135 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117607.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117607.log.gz 2023-12-25T20:56:51.135 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118609.log 2023-12-25T20:56:51.135 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118285.log 2023-12-25T20:56:51.135 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117513.log.gz 2023-12-25T20:56:51.135 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117655.log 2023-12-25T20:56:51.135 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118609.log.gz 2023-12-25T20:56:51.135 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118120.log 2023-12-25T20:56:51.136 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118285.log.gz 2023-12-25T20:56:51.136 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117655.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117655.log.gz 2023-12-25T20:56:51.136 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119024.log 2023-12-25T20:56:51.136 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117902.log 2023-12-25T20:56:51.136 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118120.log.gz 2023-12-25T20:56:51.136 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.119024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118734.log 2023-12-25T20:56:51.136 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119024.log.gz 2023-12-25T20:56:51.136 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118759.log 2023-12-25T20:56:51.137 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117902.log.gz 2023-12-25T20:56:51.137 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117875.log 2023-12-25T20:56:51.137 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118734.log.gz 2023-12-25T20:56:51.137 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118784.log 2023-12-25T20:56:51.137 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118759.log.gz 2023-12-25T20:56:51.137 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114271.log 2023-12-25T20:56:51.137 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117875.log.gz 2023-12-25T20:56:51.137 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118634.log 2023-12-25T20:56:51.138 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118784.log.gz 2023-12-25T20:56:51.138 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.114271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117809.log 2023-12-25T20:56:51.138 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114271.log.gz 2023-12-25T20:56:51.138 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118241.log 2023-12-25T20:56:51.138 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118634.log.gz 2023-12-25T20:56:51.138 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118534.log 2023-12-25T20:56:51.138 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117809.log.gz 2023-12-25T20:56:51.138 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114126.log 2023-12-25T20:56:51.139 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118241.log.gz 2023-12-25T20:56:51.139 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118709.log 2023-12-25T20:56:51.139 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118534.log.gz 2023-12-25T20:56:51.139 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.114126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118410.log 2023-12-25T20:56:51.139 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114126.log.gz 2023-12-25T20:56:51.139 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118862.log 2023-12-25T20:56:51.139 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118709.log.gz 2023-12-25T20:56:51.139 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118037.log 2023-12-25T20:56:51.139 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118410.log.gz 2023-12-25T20:56:51.140 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117338.log 2023-12-25T20:56:51.140 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118862.log.gz 2023-12-25T20:56:51.140 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118998.log 2023-12-25T20:56:51.140 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118037.log.gz 2023-12-25T20:56:51.140 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118310.log 2023-12-25T20:56:51.140 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117338.log.gz 2023-12-25T20:56:51.140 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118998.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-25T20:56:51.140 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118998.log.gz 2023-12-25T20:56:51.141 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118014.log 2023-12-25T20:56:51.141 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118310.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118310.log.gz 2023-12-25T20:56:51.141 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118559.log 2023-12-25T20:56:51.141 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118915.log 2023-12-25T20:56:51.141 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118014.log.gz 2023-12-25T20:56:51.141 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118559.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-12-25T20:56:51.142 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118559.log.gz 2023-12-25T20:56:51.142 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118219.log 2023-12-25T20:56:51.142 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118915.log.gz 2023-12-25T20:56:51.142 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117416.log 2023-12-25T20:56:51.142 INFO:teuthology.orchestra.run.smithi191.stderr: 90.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-25T20:56:51.142 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118219.log.gz 2023-12-25T20:56:51.142 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117960.log 2023-12-25T20:56:51.142 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117584.log 2023-12-25T20:56:51.142 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117416.log.gz 2023-12-25T20:56:51.143 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118335.log 2023-12-25T20:56:51.143 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117960.log.gz 2023-12-25T20:56:51.143 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118659.log 2023-12-25T20:56:51.143 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117584.log.gz 2023-12-25T20:56:51.143 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117933.log 2023-12-25T20:56:51.143 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118335.log.gz 2023-12-25T20:56:51.143 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117441.log 2023-12-25T20:56:51.143 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118659.log.gz 2023-12-25T20:56:51.143 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117933.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117933.log.gz 2023-12-25T20:56:51.144 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118095.log 2023-12-25T20:56:51.144 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117730.log 2023-12-25T20:56:51.144 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117441.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117441.log.gz 2023-12-25T20:56:51.144 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118196.log 0.0% 2023-12-25T20:56:51.144 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/ceph-client.admin.118095.log.gz 2023-12-25T20:56:51.144 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118888.log 2023-12-25T20:56:51.144 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117730.log.gz 2023-12-25T20:56:51.145 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118435.log 2023-12-25T20:56:51.145 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118196.log.gz 2023-12-25T20:56:51.145 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118888.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.118888.log.gz 2023-12-25T20:56:51.145 INFO:teuthology.orchestra.run.smithi191.stderr:--verbose -- /var/log/ceph/ceph-client.admin.118584.log 2023-12-25T20:56:51.145 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-12-25T20:56:51.145 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118435.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118435.log.gz 2023-12-25T20:56:51.149 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118684.log 2023-12-25T20:56:51.149 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118584.log.gz 2023-12-25T20:56:51.149 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118940.log 2023-12-25T20:56:51.149 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118684.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118684.log.gz 2023-12-25T20:56:51.150 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118385.log 2023-12-25T20:56:51.150 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117844.log 2023-12-25T20:56:51.150 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118940.log.gz 2023-12-25T20:56:51.150 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118263.log 2023-12-25T20:56:51.150 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118385.log.gz 2023-12-25T20:56:51.150 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117844.log.gz 2023-12-25T20:56:51.150 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117365.log 2023-12-25T20:56:51.150 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114327.log 2023-12-25T20:56:51.151 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118263.log.gz 2023-12-25T20:56:51.151 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117365.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-12-25T20:56:51.151 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117365.log.gz 2023-12-25T20:56:51.151 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.114327.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.114327.log.gz 2023-12-25T20:56:51.151 INFO:teuthology.orchestra.run.smithi191.stderr:--verbose -- /var/log/ceph/ceph-client.admin.118143.log 2023-12-25T20:56:51.151 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118973.log 2023-12-25T20:56:51.151 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117490.log 2023-12-25T20:56:51.151 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118143.log.gz 2023-12-25T20:56:51.152 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118973.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.118973.log.gz--verbose 2023-12-25T20:56:51.152 INFO:teuthology.orchestra.run.smithi191.stderr:-- /var/log/ceph/ceph-client.admin.119099.log 2023-12-25T20:56:51.152 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-25T20:56:51.152 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117490.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117490.log.gz 2023-12-25T20:56:51.152 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.119099.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-12-25T20:56:51.152 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119099.log.gz 2023-12-25T20:56:51.153 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118835.log 2023-12-25T20:56:51.153 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.118835.log: 86.2% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-25T20:56:51.153 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118835.log.gz 2023-12-25T20:56:51.153 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119074.log 2023-12-25T20:56:51.153 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118360.log 2023-12-25T20:56:51.153 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.119074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114077.log 2023-12-25T20:56:51.153 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119074.log.gz 2023-12-25T20:56:51.153 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118360.log.gz 2023-12-25T20:56:51.154 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.114077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114077.log.gz 2023-12-25T20:56:51.154 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117784.log 2023-12-25T20:56:51.154 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117632.log 2023-12-25T20:56:51.154 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117559.log 2023-12-25T20:56:51.154 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117784.log.gz 2023-12-25T20:56:51.154 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117632.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117632.log.gz 2023-12-25T20:56:51.154 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117559.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117559.log.gz 2023-12-25T20:56:51.154 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119049.log 2023-12-25T20:56:51.155 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117536.log 2023-12-25T20:56:51.155 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.119049.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119049.log.gz 2023-12-25T20:56:51.155 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117536.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117536.log.gz 2023-12-25T20:56:51.155 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-12-25T20:56:51.155 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117703.log 2023-12-25T20:56:51.155 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117757.log 2023-12-25T20:56:51.155 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117703.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117703.log.gz 2023-12-25T20:56:51.155 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117757.log.gz 2023-12-25T20:56:51.155 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114224.log 2023-12-25T20:56:51.156 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114174.log 2023-12-25T20:56:51.156 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.114224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114224.log.gz 2023-12-25T20:56:51.156 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.114174.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114174.log.gz 2023-12-25T20:56:51.156 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118173.log 2023-12-25T20:56:51.156 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117681.log 2023-12-25T20:56:51.156 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118173.log.gz 2023-12-25T20:56:51.156 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117681.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117681.log.gz 2023-12-25T20:56:51.156 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117390.log 2023-12-25T20:56:51.157 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118063.log 2023-12-25T20:56:51.157 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117390.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117390.log.gz 2023-12-25T20:56:51.157 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118063.log.gz 2023-12-25T20:56:51.157 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117991.log 2023-12-25T20:56:51.157 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117467.log 2023-12-25T20:56:51.157 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117991.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117991.log.gz 2023-12-25T20:56:51.157 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.117467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117467.log.gz 2023-12-25T20:56:51.157 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118509.log 2023-12-25T20:56:51.158 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.118509.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118509.log.gz 2023-12-25T20:56:51.158 INFO:teuthology.orchestra.run.smithi191.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-25T20:56:51.165 INFO:teuthology.orchestra.run.smithi153.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-12-25T20:56:51.221 INFO:teuthology.orchestra.run.smithi191.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-12-25T20:56:51.223 INFO:teuthology.orchestra.run.smithi153.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-25T20:56:51.626 INFO:teuthology.orchestra.run.smithi153.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-25T20:56:51.656 INFO:teuthology.orchestra.run.smithi153.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-25T20:56:51.669 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-12-25T20:56:51.684 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-12-25T20:56:51.693 INFO:teuthology.orchestra.run.smithi153.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-25T20:56:51.694 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-12-25T20:56:51.694 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m0.653s 2023-12-25T20:56:51.695 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m1.986s 2023-12-25T20:56:51.695 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m0.123s 2023-12-25T20:56:51.706 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-12-25T20:56:51.708 INFO:teuthology.orchestra.run.smithi191.stderr: 2023-12-25T20:56:51.708 INFO:teuthology.orchestra.run.smithi191.stderr:real 0m0.628s 2023-12-25T20:56:51.708 INFO:teuthology.orchestra.run.smithi191.stderr:user 0m1.875s 2023-12-25T20:56:51.708 INFO:teuthology.orchestra.run.smithi191.stderr:sys 0m0.103s 2023-12-25T20:56:51.708 INFO:tasks.ceph:Archiving logs... 2023-12-25T20:56:51.709 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500952/remote/smithi153/log 2023-12-25T20:56:51.710 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-25T20:56:52.157 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500952/remote/smithi191/log 2023-12-25T20:56:52.158 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-25T20:56:52.548 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-25T20:56:52.558 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-25T20:56:52.558 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-25T20:56:52.561 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-25T20:56:52.640 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-25T20:56:52.684 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-25T20:56:52.685 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-25T20:56:52.687 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-25T20:56:52.702 INFO:teuthology.orchestra.run.smithi153.stderr:bash: ntpq: command not found 2023-12-25T20:56:52.706 INFO:teuthology.orchestra.run.smithi191.stderr:bash: ntpq: command not found 2023-12-25T20:56:52.706 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-25T20:56:52.706 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2023-12-25T20:56:52.707 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 +368us[ +368us] +/- 83ms 2023-12-25T20:56:52.707 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 +152us[ +179us] +/- 75ms 2023-12-25T20:56:52.707 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 -539us[ -512us] +/- 92ms 2023-12-25T20:56:52.707 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T20:56:52.708 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-25T20:56:52.709 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2023-12-25T20:56:52.709 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 +374us[ +374us] +/- 83ms 2023-12-25T20:56:52.709 INFO:teuthology.orchestra.run.smithi191.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 +134us[ +141us] +/- 74ms 2023-12-25T20:56:52.709 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 -543us[ -543us] +/- 92ms 2023-12-25T20:56:52.709 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T20:56:52.712 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-25T20:56:52.754 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-25T20:56:52.756 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-25T20:56:52.785 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-25T20:56:52.876 DEBUG:teuthology.orchestra.run.smithi191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-25T20:56:52.965 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-25T20:56:52.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T20:56:52.998 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-25T20:56:53.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T20:56:53.030 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-25T20:56:53.039 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-25T20:56:53.056 INFO:teuthology.task.internal:Duration was 998.712548 seconds 2023-12-25T20:56:53.057 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-25T20:56:53.088 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-25T20:56:53.088 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-25T20:56:53.092 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-25T20:56:53.136 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-25T20:56:53.139 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-25T20:56:53.462 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-25T20:56:53.463 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2023-12-25T20:56:53.463 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-25T20:56:53.485 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2023-12-25T20:56:53.485 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-25T20:56:53.510 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-25T20:56:53.510 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-25T20:56:53.528 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-25T20:56:53.587 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-25T20:56:53.588 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-25T20:56:53.607 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-25T20:56:53.693 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-25T20:56:53.707 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-25T20:56:53.707 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-25T20:56:53.734 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-25T20:56:53.761 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-25T20:56:53.771 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-25T20:56:53.776 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-25T20:56:53.801 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2023-12-25T20:56:53.828 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2023-12-25T20:56:53.870 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-25T20:56:53.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T20:56:53.888 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-25T20:56:53.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T20:56:53.928 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-25T20:56:53.939 INFO:teuthology.task.internal:Transferring archived files... 2023-12-25T20:56:53.940 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500952/remote/smithi153 2023-12-25T20:56:53.940 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-25T20:56:53.979 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500952/remote/smithi191 2023-12-25T20:56:53.980 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-25T20:56:54.020 INFO:teuthology.task.internal:Removing archive directory... 2023-12-25T20:56:54.021 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-25T20:56:54.023 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-25T20:56:54.068 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-25T20:56:54.080 INFO:teuthology.task.internal:Not uploading archives. 2023-12-25T20:56:54.080 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-25T20:56:54.194 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-25T20:56:54.194 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-25T20:56:54.197 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-25T20:56:54.212 INFO:teuthology.orchestra.run.smithi153.stdout: 525833 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 25 20:56 /home/ubuntu/cephtest 2023-12-25T20:56:54.214 INFO:teuthology.orchestra.run.smithi191.stdout: 525359 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 25 20:56 /home/ubuntu/cephtest 2023-12-25T20:56:54.214 INFO:teuthology.orchestra.run.smithi191.stdout: 920546 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Dec 25 20:52 /home/ubuntu/cephtest/mnt.0 2023-12-25T20:56:54.214 INFO:teuthology.orchestra.run.smithi191.stdout: 920547 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Dec 25 20:55 /home/ubuntu/cephtest/mnt.0/client.0 2023-12-25T20:56:54.214 INFO:teuthology.orchestra.run.smithi191.stdout: 920588 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Dec 25 20:55 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-25T20:56:54.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T20:56:54.215 INFO:teuthology.orchestra.run.smithi191.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-12-25T20:56:54.216 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi191 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-12-25T20:56:54.233 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-25T20:56:54.246 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-25T20:56:54.265 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi191 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-12-25T20:56:54.266 INFO:teuthology.nuke:Checking targets against current locks 2023-12-25T20:56:54.293 DEBUG:teuthology.nuke:shortname: smithi153 2023-12-25T20:56:54.294 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-25T20:56:54.317 DEBUG:teuthology.nuke:shortname: smithi191 2023-12-25T20:56:54.317 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-25T20:56:54.321 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500952', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-25 20:31:35.172889', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHwsX+5nIByVAfAQ8I7l1bnFaNLiDHwJYt5U5Zvs8zKhl42cGlAb4liT5eIqnXZhTjMK80xEHH1sp+lHie9P4w4='} 2023-12-25T20:56:54.346 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500952', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-25 20:31:35.174303', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD+ISzGpHk1nnk50JTFdMJHaU3XrYxVp8J6UrAV6J4AP0mz5orSF5TRpCHUvF8HD2drq6QTywF6azXOm3Jow5dE='} 2023-12-25T20:56:54.379 INFO:teuthology.orchestra.console.smithi153:Power off 2023-12-25T20:56:54.379 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-25T20:56:54.404 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off 2023-12-25T20:56:54.404 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-25T20:56:54.424 INFO:teuthology.orchestra.console.smithi191:Power off 2023-12-25T20:56:54.425 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-25T20:56:54.440 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2023-12-25T20:56:54.454 DEBUG:teuthology.orchestra.console.smithi191:power off output: Chassis Power Control: Down/Off 2023-12-25T20:56:54.454 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-25T20:56:54.488 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2023-12-25T20:56:58.441 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-25T20:56:58.489 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-25T20:56:58.554 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2023-12-25T20:56:58.601 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2023-12-25T20:57:02.557 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-25T20:57:02.603 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-25T20:57:02.668 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2023-12-25T20:57:02.715 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2023-12-25T20:57:06.669 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-25T20:57:06.716 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-25T20:57:06.780 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off 2023-12-25T20:57:06.827 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is off 2023-12-25T20:57:06.881 INFO:teuthology.orchestra.console.smithi153:Power off completed 2023-12-25T20:57:06.928 INFO:teuthology.orchestra.console.smithi191:Power off completed 2023-12-25T20:57:07.035 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/one workloads/rados_mon_workunits} duration: 998.7125475406647 failure_reason: 'Command failed (workunit test mon/crush_ops.sh) on smithi191 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=d84aeaee14174e399a73444e7021da86 status: fail success: false 2023-12-25T20:57:07.035 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-25T20:57:07.120 INFO:teuthology.run:FAIL