2024-04-25T19:36:55.122 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T19:36:55.122 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/pdonnell-2024-04-25_11:25:24-rados:mgr-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673402 --description rados:mgr/{clusters/{2-node-mgr} debug/mgr distro/{centos_latest} mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-hybrid} tasks/{1-install 2-ceph 3-mgrmodules 4-units/workunits}} --owner scheduled_pdonnell@teuthology --verbose --name pdonnell-2024-04-25_11:25:24-rados:mgr-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi -- /home/teuthworker/archive/pdonnell-2024-04-25_11:25:24-rados:mgr-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673402/orig.config.yaml 2024-04-25T19:36:55.156 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T19:36:55.238 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-04-25_11:25:24-rados:mgr-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673402 branch: wip-pdonnell-testing-20240425.015853-debug description: rados:mgr/{clusters/{2-node-mgr} debug/mgr distro/{centos_latest} mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-hybrid} tasks/{1-install 2-ceph 3-mgrmodules 4-units/workunits}} email: pdonnell@redhat.com first_in_suite: false job_id: '7673402' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-04-25_11:25:24-rados:mgr-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 2 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240425.015853-debug ceph: conf: client: debug client: 20 debug mgrc: 20 debug ms: 1 global: mon election default strategy: 1 mds: debug mgrc: 20 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mgr ttl cache expire seconds: 0 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug mgrc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240425.015853-debug sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 owner: scheduled_pdonnell@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mgr.x - mon.a - mon.c - mds.a - mds.c - osd.0 - client.0 - - mgr.y - mgr.z - mon.b - mds.b - osd.1 - osd.2 - osd.3 - client.1 seed: 5640 sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 sleep_before_teardown: 0 suite: rados:mgr suite_branch: wip-pdonnell-testing-20240425.015853-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFTQf+YLUeZx3Qzl6VT2y+2b6pyxkwYpUb7FnJerV+A9TCdXrtKOhp/EXR8H9KBCwKURpDh6lM4swsNxSA9OY/M= smithi102.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJWFpWP0AIgZgBUgvEzasqXppOmczEePF8QeyxB5Dw5gLBggdtMDyxT/pr3HSP7+6836gvoDMHiZMJL5uaLsFn4= tasks: - install: null - ceph: log-ignorelist: - overall HEALTH_ - \(MGR_DOWN\) - \(PG_ - replacing it with standby - No standby daemons available - \(POOL_APP_NOT_ENABLED\) wait-for-scrub: false - sequential: - mgrmodules - workunit: clients: client.0: - mgr/test_localpool.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_11:25:24 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T19:36:55.238 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa; will attempt to use it 2024-04-25T19:36:55.239 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks 2024-04-25T19:36:55.239 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T19:36:55.241 INFO:teuthology.task.internal:Checking packages... 2024-04-25T19:36:55.266 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '994e2f20102a20fa9a27966c14b625cc01884df8' 2024-04-25T19:36:55.266 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T19:36:55.267 INFO:teuthology.packaging:ref: None 2024-04-25T19:36:55.267 INFO:teuthology.packaging:tag: None 2024-04-25T19:36:55.267 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240425.015853-debug 2024-04-25T19:36:55.267 INFO:teuthology.packaging:sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T19:36:55.267 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240425.015853-debug 2024-04-25T19:36:55.490 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3291.g994e2f20 2024-04-25T19:36:55.492 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T19:36:55.498 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T19:36:55.498 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T19:36:55.506 INFO:teuthology.task.internal:Saving configuration 2024-04-25T19:36:55.520 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T19:36:55.527 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T19:36:55.551 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_11:25:24-rados:mgr-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673402', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 19:31:44.193243', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFTQf+YLUeZx3Qzl6VT2y+2b6pyxkwYpUb7FnJerV+A9TCdXrtKOhp/EXR8H9KBCwKURpDh6lM4swsNxSA9OY/M='} 2024-04-25T19:36:55.570 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_11:25:24-rados:mgr-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673402', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 19:31:44.194580', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJWFpWP0AIgZgBUgvEzasqXppOmczEePF8QeyxB5Dw5gLBggdtMDyxT/pr3HSP7+6836gvoDMHiZMJL5uaLsFn4='} 2024-04-25T19:36:55.570 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T19:36:55.578 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2024-04-25T19:36:55.578 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2024-04-25T19:36:55.578 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T19:36:55.654 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4f0d285e50>, signals=[15]) 2024-04-25T19:36:55.655 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T19:36:55.661 INFO:teuthology.task.internal:Opening connections... 2024-04-25T19:36:55.661 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2024-04-25T19:36:55.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T19:36:55.743 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T19:36:55.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T19:36:55.818 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T19:36:55.824 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2024-04-25T19:36:55.842 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2024-04-25T19:36:55.843 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2024-04-25T19:36:55.901 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2024-04-25T19:36:55.901 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="9" 2024-04-25T19:36:55.901 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2024-04-25T19:36:55.901 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2024-04-25T19:36:55.901 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="9" 2024-04-25T19:36:55.901 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el9" 2024-04-25T19:36:55.901 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T19:36:55.901 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2024-04-25T19:36:55.901 INFO:teuthology.orchestra.run.smithi029.stdout:LOGO="fedora-logo-icon" 2024-04-25T19:36:55.901 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T19:36:55.902 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2024-04-25T19:36:55.902 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T19:36:55.902 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T19:36:55.902 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T19:36:55.902 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2024-04-25T19:36:55.924 DEBUG:teuthology.orchestra.run.smithi102:> uname -m 2024-04-25T19:36:55.941 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64 2024-04-25T19:36:55.941 DEBUG:teuthology.orchestra.run.smithi102:> cat /etc/os-release 2024-04-25T19:36:55.998 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="CentOS Stream" 2024-04-25T19:36:55.998 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="9" 2024-04-25T19:36:55.998 INFO:teuthology.orchestra.run.smithi102.stdout:ID="centos" 2024-04-25T19:36:55.998 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE="rhel fedora" 2024-04-25T19:36:55.998 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="9" 2024-04-25T19:36:55.998 INFO:teuthology.orchestra.run.smithi102.stdout:PLATFORM_ID="platform:el9" 2024-04-25T19:36:55.998 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T19:36:55.998 INFO:teuthology.orchestra.run.smithi102.stdout:ANSI_COLOR="0;31" 2024-04-25T19:36:55.998 INFO:teuthology.orchestra.run.smithi102.stdout:LOGO="fedora-logo-icon" 2024-04-25T19:36:55.998 INFO:teuthology.orchestra.run.smithi102.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T19:36:55.999 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://centos.org/" 2024-04-25T19:36:55.999 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T19:36:55.999 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T19:36:55.999 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T19:36:55.999 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2024-04-25T19:36:56.024 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T19:36:56.032 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T19:36:56.038 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T19:36:56.038 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2024-04-25T19:36:56.041 DEBUG:teuthology.orchestra.run.smithi102:> test '!' -e /home/ubuntu/cephtest 2024-04-25T19:36:56.057 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T19:36:56.063 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T19:36:56.063 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2024-04-25T19:36:56.098 DEBUG:teuthology.orchestra.run.smithi102:> test -z $(ls -A /var/lib/ceph) 2024-04-25T19:36:56.204 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T19:36:56.257 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T19:36:56.273 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-25T19:36:56.273 INFO:teuthology.task.kernel:config {'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-25T19:36:56.273 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T19:36:56.274 DEBUG:teuthology.orchestra.run.smithi102:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T19:36:56.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:36:56.290 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-04-25T19:36:56.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:36:56.291 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-04-25T19:36:56.348 INFO:teuthology.orchestra.run.smithi102.stdout:5.14.0-437.el9.x86_64 2024-04-25T19:36:56.348 INFO:teuthology.task.kernel:Running kernel on smithi102: 5.14.0-437.el9.x86_64 2024-04-25T19:36:56.348 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum install -y kernel 2024-04-25T19:36:56.349 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-437.el9.x86_64 2024-04-25T19:36:56.350 INFO:teuthology.task.kernel:Running kernel on smithi029: 5.14.0-437.el9.x86_64 2024-04-25T19:36:56.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum install -y kernel 2024-04-25T19:36:59.249 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 6.3 MB/s | 8.0 MB 00:01 2024-04-25T19:36:59.702 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 9.2 MB/s | 8.0 MB 00:00 2024-04-25T19:37:01.167 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-04-25T19:37:01.722 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-25T19:37:05.476 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 6.2 MB 00:00 2024-04-25T19:37:06.124 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.2 MB 00:00 2024-04-25T19:37:07.024 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-04-25T19:37:07.815 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-25T19:37:08.876 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-25T19:37:08.988 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-04-25T19:37:13.875 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-25T19:37:14.058 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:01 ago on Thu 25 Apr 2024 07:37:13 PM UTC. 2024-04-25T19:37:14.287 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-25T19:37:15.765 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T19:37:15.765 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T19:37:15.866 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:37:15.867 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-04-25T19:37:15.867 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:37:15.973 DEBUG:teuthology.orchestra.run.smithi102:> echo no | sudo yum reinstall kernel || true 2024-04-25T19:37:16.277 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T19:37:16.278 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T19:37:16.397 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:37:16.398 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-25T19:37:16.399 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:37:16.471 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 07:37:13 PM UTC. 2024-04-25T19:37:16.525 DEBUG:teuthology.orchestra.run.smithi029:> echo no | sudo yum reinstall kernel || true 2024-04-25T19:37:16.597 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:37:16.606 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:37:16.606 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-04-25T19:37:16.607 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:37:16.607 INFO:teuthology.orchestra.run.smithi102.stdout:Reinstalling: 2024-04-25T19:37:16.607 INFO:teuthology.orchestra.run.smithi102.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T19:37:16.607 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:37:16.607 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-04-25T19:37:16.607 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:37:16.607 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:37:16.608 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 255 k 2024-04-25T19:37:16.608 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 0 2024-04-25T19:37:16.608 INFO:teuthology.orchestra.run.smithi102.stderr:Operation aborted. 2024-04-25T19:37:16.661 INFO:teuthology.orchestra.run.smithi102.stdout:Is this ok [y/N]: 2024-04-25T19:37:16.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum reinstall -y kernel || true 2024-04-25T19:37:17.048 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 07:37:14 PM UTC. 2024-04-25T19:37:17.154 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:04 ago on Thu 25 Apr 2024 07:37:13 PM UTC. 2024-04-25T19:37:17.190 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:37:17.216 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:37:17.216 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-04-25T19:37:17.216 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:37:17.216 INFO:teuthology.orchestra.run.smithi029.stdout:Reinstalling: 2024-04-25T19:37:17.216 INFO:teuthology.orchestra.run.smithi029.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T19:37:17.216 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:37:17.216 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-25T19:37:17.216 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:37:17.216 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:37:17.217 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 255 k 2024-04-25T19:37:17.217 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 0 2024-04-25T19:37:17.217 INFO:teuthology.orchestra.run.smithi029.stderr:Operation aborted. 2024-04-25T19:37:17.269 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:37:17.270 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:37:17.270 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-04-25T19:37:17.270 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:37:17.270 INFO:teuthology.orchestra.run.smithi102.stdout:Reinstalling: 2024-04-25T19:37:17.270 INFO:teuthology.orchestra.run.smithi102.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T19:37:17.270 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:37:17.270 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-04-25T19:37:17.270 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:37:17.270 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:37:17.271 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 255 k 2024-04-25T19:37:17.271 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 0 2024-04-25T19:37:17.271 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-04-25T19:37:17.278 INFO:teuthology.orchestra.run.smithi029.stdout:Is this ok [y/N]: 2024-04-25T19:37:17.278 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum reinstall -y kernel || true 2024-04-25T19:37:17.686 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.8 MB/s | 255 kB 00:00 2024-04-25T19:37:17.686 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:37:17.687 INFO:teuthology.orchestra.run.smithi102.stdout:Total 612 kB/s | 255 kB 00:00 2024-04-25T19:37:17.702 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-04-25T19:37:17.792 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 07:37:14 PM UTC. 2024-04-25T19:37:17.885 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-04-25T19:37:17.886 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-04-25T19:37:17.890 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-04-25T19:37:17.891 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-04-25T19:37:17.913 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:37:17.914 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:37:17.914 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-04-25T19:37:17.914 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:37:17.914 INFO:teuthology.orchestra.run.smithi029.stdout:Reinstalling: 2024-04-25T19:37:17.914 INFO:teuthology.orchestra.run.smithi029.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T19:37:17.914 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:37:17.914 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-25T19:37:17.914 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:37:17.914 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:37:17.914 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 255 k 2024-04-25T19:37:17.915 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 0 2024-04-25T19:37:17.915 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-25T19:37:18.037 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-04-25T19:37:18.120 INFO:teuthology.orchestra.run.smithi102.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T19:37:18.120 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T19:37:18.191 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.9 MB/s | 255 kB 00:00 2024-04-25T19:37:18.192 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:37:18.192 INFO:teuthology.orchestra.run.smithi029.stdout:Total 918 kB/s | 255 kB 00:00 2024-04-25T19:37:18.209 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-25T19:37:18.420 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-25T19:37:18.420 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-25T19:37:18.425 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-25T19:37:18.425 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-25T19:37:18.599 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-25T19:37:18.711 INFO:teuthology.orchestra.run.smithi029.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T19:37:18.711 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T19:37:18.803 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T19:37:18.803 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T19:37:19.549 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T19:37:19.549 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:37:19.549 INFO:teuthology.orchestra.run.smithi102.stdout:Reinstalled: 2024-04-25T19:37:19.549 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T19:37:19.549 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:37:19.550 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:37:19.719 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T19:37:19.760 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T19:37:19.760 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T19:37:19.764 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T19:37:19.765 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T19:37:19.765 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T19:37:19.765 INFO:teuthology.task.kernel:Enabling kdb on mgr.y... 2024-04-25T19:37:19.766 DEBUG:teuthology.orchestra.run.smithi102:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T19:37:19.803 INFO:teuthology.orchestra.run.smithi102.stdout:ttyS1 2024-04-25T19:37:19.832 DEBUG:teuthology.parallel:result is None 2024-04-25T19:37:20.619 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T19:37:20.619 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:37:20.619 INFO:teuthology.orchestra.run.smithi029.stdout:Reinstalled: 2024-04-25T19:37:20.619 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T19:37:20.619 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:37:20.619 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:37:20.835 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T19:37:20.883 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T19:37:20.883 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T19:37:20.883 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T19:37:20.883 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-04-25T19:37:20.884 DEBUG:teuthology.orchestra.run.smithi029:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T19:37:20.917 INFO:teuthology.orchestra.run.smithi029.stdout:ttyS1 2024-04-25T19:37:20.944 DEBUG:teuthology.parallel:result is None 2024-04-25T19:37:20.945 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T19:37:20.951 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T19:37:20.952 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T19:37:20.987 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T19:37:21.005 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T19:37:21.011 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T19:37:21.018 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T19:37:21.018 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T19:37:21.043 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T19:37:21.077 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T19:37:21.084 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T19:37:21.084 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T19:37:21.103 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T19:37:21.134 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T19:37:21.143 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T19:37:21.151 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T19:37:21.162 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T19:37:21.163 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T19:37:21.170 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T19:37:21.170 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T19:37:21.188 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T19:37:21.229 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T19:37:21.240 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T19:37:21.240 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T19:37:21.253 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T19:37:21.285 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T19:37:21.339 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T19:37:21.395 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T19:37:21.479 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T19:37:21.516 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-25T19:37:21.517 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T19:37:21.581 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T19:37:21.624 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T19:37:21.672 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T19:37:21.755 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T19:37:21.796 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T19:37:21.796 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T19:37:21.860 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2024-04-25T19:37:21.862 DEBUG:teuthology.orchestra.run.smithi102:> sudo service rsyslog restart 2024-04-25T19:37:21.899 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T19:37:21.938 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T19:37:22.393 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T19:37:22.407 INFO:teuthology.task.internal:Starting timer... 2024-04-25T19:37:22.407 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T19:37:22.438 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T19:37:22.447 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-25T19:37:22.448 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2024-04-25T19:37:22.534 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2024-04-25T19:37:22.536 DEBUG:teuthology.orchestra.run.smithi102:> sudo service auditd rotate 2024-04-25T19:37:22.583 INFO:teuthology.orchestra.run.smithi102.stdout:Rotating logs: 2024-04-25T19:37:22.585 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T19:37:22.585 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2024-04-25T19:37:22.607 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2024-04-25T19:37:22.607 DEBUG:teuthology.orchestra.run.smithi102:> /usr/sbin/getenforce 2024-04-25T19:37:22.633 INFO:teuthology.orchestra.run.smithi102.stdout:Permissive 2024-04-25T19:37:22.633 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi102.front.sepia.ceph.com': 'permissive'} 2024-04-25T19:37:22.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T19:37:22.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:37:22.672 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T19:37:22.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:37:22.697 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T19:37:22.697 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2024-04-25T19:37:22.744 DEBUG:teuthology.orchestra.run.smithi102:> sudo /usr/sbin/setenforce permissive 2024-04-25T19:37:22.768 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T19:37:22.777 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-25T19:37:22.782 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-25T19:37:22.944 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T19:37:22.961 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T19:37:22.962 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi102.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T19:40:59.222 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi102.front.sepia.ceph.com')] 2024-04-25T19:40:59.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-04-25T19:40:59.225 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T19:40:59.300 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-04-25T19:40:59.386 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-04-25T19:40:59.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-04-25T19:40:59.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T19:40:59.462 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-04-25T19:40:59.537 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-04-25T19:40:59.538 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T19:40:59.549 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T19:40:59.549 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T19:40:59.549 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T19:40:59.552 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T19:40:59.552 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T19:40:59.580 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T19:40:59.594 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T19:40:59.605 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T19:40:59.618 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T19:40:59.618 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2024-04-25T19:40:59.629 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2024-04-25T19:40:59.641 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: ntpd: command not found 2024-04-25T19:40:59.643 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T19:40:59.650 INFO:teuthology.orchestra.run.smithi102.stdout:506 Cannot talk to daemon 2024-04-25T19:40:59.657 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T19:40:59.663 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T19:40:59.676 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T19:40:59.711 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-04-25T19:40:59.714 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T19:40:59.714 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-04-25T19:40:59.714 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:40:59.714 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:40:59.714 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:40:59.714 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:40:59.732 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-04-25T19:40:59.735 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T19:40:59.735 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-04-25T19:40:59.736 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:40:59.736 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:40:59.736 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:40:59.736 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:40:59.737 INFO:teuthology.run_tasks:Running task install... 2024-04-25T19:40:59.745 DEBUG:teuthology.task.install:project ceph 2024-04-25T19:40:59.745 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}} 2024-04-25T19:40:59.745 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'} 2024-04-25T19:40:59.745 INFO:teuthology.task.install:Using flavor: default 2024-04-25T19:40:59.751 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T19:40:59.751 INFO:teuthology.task.install:extra packages: [] 2024-04-25T19:40:59.752 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T19:40:59.752 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T19:40:59.754 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T19:40:59.754 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T19:40:59.973 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T19:40:59.973 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T19:40:59.981 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T19:40:59.981 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T19:41:00.081 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T19:41:00.082 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-25T19:41:00.082 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T19:41:00.088 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T19:41:00.089 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T19:41:00.089 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T19:41:00.109 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-25T19:41:00.109 DEBUG:teuthology.orchestra.run.smithi029:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T19:41:00.113 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-25T19:41:00.113 DEBUG:teuthology.orchestra.run.smithi102:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T19:41:00.176 DEBUG:teuthology.orchestra.run.smithi029:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T19:41:00.177 DEBUG:teuthology.orchestra.run.smithi102:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T19:41:00.251 DEBUG:teuthology.orchestra.run.smithi102:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T19:41:00.252 DEBUG:teuthology.orchestra.run.smithi029:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T19:41:00.314 INFO:teuthology.orchestra.run.smithi102.stdout:check_obsoletes = 1 2024-04-25T19:41:00.316 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum clean all 2024-04-25T19:41:00.318 INFO:teuthology.orchestra.run.smithi029.stdout:check_obsoletes = 1 2024-04-25T19:41:00.319 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum clean all 2024-04-25T19:41:00.538 INFO:teuthology.orchestra.run.smithi102.stdout:48 files removed 2024-04-25T19:41:00.554 INFO:teuthology.orchestra.run.smithi029.stdout:48 files removed 2024-04-25T19:41:00.565 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-radosgw 2024-04-25T19:41:00.582 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw 2024-04-25T19:41:01.167 INFO:teuthology.orchestra.run.smithi102.stdout:ceph packages for x86_64 240 kB/s | 84 kB 00:00 2024-04-25T19:41:01.258 INFO:teuthology.orchestra.run.smithi029.stdout:ceph packages for x86_64 207 kB/s | 84 kB 00:00 2024-04-25T19:41:01.685 INFO:teuthology.orchestra.run.smithi102.stdout:ceph noarch packages 25 kB/s | 12 kB 00:00 2024-04-25T19:41:01.775 INFO:teuthology.orchestra.run.smithi029.stdout:ceph noarch packages 25 kB/s | 12 kB 00:00 2024-04-25T19:41:02.035 INFO:teuthology.orchestra.run.smithi102.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-04-25T19:41:02.151 INFO:teuthology.orchestra.run.smithi029.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-04-25T19:41:02.655 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-25T19:41:02.813 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-25T19:41:04.807 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-04-25T19:41:04.993 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-04-25T19:41:08.848 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-04-25T19:41:09.286 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-25T19:41:10.505 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-04-25T19:41:10.987 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-04-25T19:41:12.626 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-25T19:41:12.856 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 9.4 MB/s | 21 MB 00:02 2024-04-25T19:41:17.900 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-25T19:41:17.915 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-04-25T19:41:20.103 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:41:20.103 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:41:20.103 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-04-25T19:41:20.103 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:41:20.103 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-04-25T19:41:20.103 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T19:41:20.103 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrading: 2024-04-25T19:41:20.104 INFO:teuthology.orchestra.run.smithi102.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T19:41:20.104 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T19:41:20.104 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-04-25T19:41:20.104 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T19:41:20.104 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T19:41:20.104 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T19:41:20.104 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T19:41:20.104 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T19:41:20.104 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T19:41:20.104 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T19:41:20.104 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T19:41:20.105 INFO:teuthology.orchestra.run.smithi102.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T19:41:20.105 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T19:41:20.105 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T19:41:20.105 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T19:41:20.105 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T19:41:20.105 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T19:41:20.105 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T19:41:20.105 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T19:41:20.105 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T19:41:20.105 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T19:41:20.105 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T19:41:20.105 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T19:41:20.105 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T19:41:20.106 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T19:41:20.106 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T19:41:20.106 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T19:41:20.106 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T19:41:20.106 INFO:teuthology.orchestra.run.smithi102.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T19:41:20.106 INFO:teuthology.orchestra.run.smithi102.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T19:41:20.106 INFO:teuthology.orchestra.run.smithi102.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T19:41:20.106 INFO:teuthology.orchestra.run.smithi102.stdout:Installing weak dependencies: 2024-04-25T19:41:20.106 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T19:41:20.106 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:41:20.106 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-04-25T19:41:20.107 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:41:20.107 INFO:teuthology.orchestra.run.smithi102.stdout:Install 30 Packages 2024-04-25T19:41:20.107 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrade 2 Packages 2024-04-25T19:41:20.107 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:41:20.107 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 58 M 2024-04-25T19:41:20.107 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-04-25T19:41:20.195 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:41:20.198 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:41:20.198 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-25T19:41:20.198 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:41:20.198 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-25T19:41:20.198 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T19:41:20.198 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2024-04-25T19:41:20.198 INFO:teuthology.orchestra.run.smithi029.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T19:41:20.198 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T19:41:20.198 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-25T19:41:20.199 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T19:41:20.199 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T19:41:20.199 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T19:41:20.199 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T19:41:20.199 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T19:41:20.199 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T19:41:20.199 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T19:41:20.199 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T19:41:20.199 INFO:teuthology.orchestra.run.smithi029.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T19:41:20.199 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T19:41:20.199 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T19:41:20.199 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T19:41:20.199 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T19:41:20.200 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T19:41:20.200 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T19:41:20.200 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T19:41:20.200 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T19:41:20.200 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T19:41:20.200 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T19:41:20.200 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T19:41:20.200 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T19:41:20.200 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T19:41:20.200 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T19:41:20.200 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T19:41:20.200 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T19:41:20.200 INFO:teuthology.orchestra.run.smithi029.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T19:41:20.200 INFO:teuthology.orchestra.run.smithi029.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T19:41:20.201 INFO:teuthology.orchestra.run.smithi029.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T19:41:20.201 INFO:teuthology.orchestra.run.smithi029.stdout:Installing weak dependencies: 2024-04-25T19:41:20.201 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T19:41:20.201 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:41:20.201 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-25T19:41:20.201 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:41:20.201 INFO:teuthology.orchestra.run.smithi029.stdout:Install 30 Packages 2024-04-25T19:41:20.201 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 2 Packages 2024-04-25T19:41:20.201 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:41:20.202 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 58 M 2024-04-25T19:41:20.202 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-25T19:41:23.383 INFO:teuthology.orchestra.run.smithi102.stdout:(1/32): ceph-base-19.0.0-3291.g994e2f20.el9.x86 1.9 MB/s | 5.4 MB 00:02 2024-04-25T19:41:23.466 INFO:teuthology.orchestra.run.smithi102.stdout:(2/32): ceph-selinux-19.0.0-3291.g994e2f20.el9. 303 kB/s | 25 kB 00:00 2024-04-25T19:41:23.800 INFO:teuthology.orchestra.run.smithi102.stdout:(3/32): libcephfs2-19.0.0-3291.g994e2f20.el9.x8 2.2 MB/s | 756 kB 00:00 2024-04-25T19:41:24.067 INFO:teuthology.orchestra.run.smithi102.stdout:(4/32): libradosstriper1-19.0.0-3291.g994e2f20. 1.8 MB/s | 497 kB 00:00 2024-04-25T19:41:25.011 INFO:teuthology.orchestra.run.smithi029.stdout:(1/32): ceph-base-19.0.0-3291.g994e2f20.el9.x86 1.2 MB/s | 5.4 MB 00:04 2024-04-25T19:41:25.061 INFO:teuthology.orchestra.run.smithi029.stdout:(2/32): ceph-selinux-19.0.0-3291.g994e2f20.el9. 506 kB/s | 25 kB 00:00 2024-04-25T19:41:25.278 INFO:teuthology.orchestra.run.smithi029.stdout:(3/32): libcephfs2-19.0.0-3291.g994e2f20.el9.x8 3.4 MB/s | 756 kB 00:00 2024-04-25T19:41:25.437 INFO:teuthology.orchestra.run.smithi029.stdout:(4/32): libradosstriper1-19.0.0-3291.g994e2f20. 3.1 MB/s | 497 kB 00:00 2024-04-25T19:41:25.550 INFO:teuthology.orchestra.run.smithi102.stdout:(5/32): librgw2-19.0.0-3291.g994e2f20.el9.x86_6 3.5 MB/s | 5.2 MB 00:01 2024-04-25T19:41:25.592 INFO:teuthology.orchestra.run.smithi102.stdout:(6/32): python3-ceph-argparse-19.0.0-3291.g994e 1.1 MB/s | 45 kB 00:00 2024-04-25T19:41:25.642 INFO:teuthology.orchestra.run.smithi102.stdout:(7/32): python3-ceph-common-19.0.0-3291.g994e2f 2.6 MB/s | 135 kB 00:00 2024-04-25T19:41:25.709 INFO:teuthology.orchestra.run.smithi102.stdout:(8/32): python3-cephfs-19.0.0-3291.g994e2f20.el 2.4 MB/s | 161 kB 00:00 2024-04-25T19:41:25.810 INFO:teuthology.orchestra.run.smithi102.stdout:(9/32): python3-rados-19.0.0-3291.g994e2f20.el9 3.1 MB/s | 321 kB 00:00 2024-04-25T19:41:25.887 INFO:teuthology.orchestra.run.smithi102.stdout:(10/32): python3-rbd-19.0.0-3291.g994e2f20.el9. 3.9 MB/s | 298 kB 00:00 2024-04-25T19:41:25.935 INFO:teuthology.orchestra.run.smithi102.stdout:(11/32): python3-rgw-19.0.0-3291.g994e2f20.el9. 1.9 MB/s | 100 kB 00:00 2024-04-25T19:41:26.044 INFO:teuthology.orchestra.run.smithi102.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 990 kB/s | 107 kB 00:00 2024-04-25T19:41:26.094 INFO:teuthology.orchestra.run.smithi102.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-25T19:41:26.150 INFO:teuthology.orchestra.run.smithi102.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 905 kB/s | 45 kB 00:00 2024-04-25T19:41:26.206 INFO:teuthology.orchestra.run.smithi102.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-04-25T19:41:26.253 INFO:teuthology.orchestra.run.smithi102.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-25T19:41:26.287 INFO:teuthology.orchestra.run.smithi102.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 42 kB 00:00 2024-04-25T19:41:26.329 INFO:teuthology.orchestra.run.smithi102.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 221 kB 00:00 2024-04-25T19:41:26.362 INFO:teuthology.orchestra.run.smithi102.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.1 MB/s | 36 kB 00:00 2024-04-25T19:41:26.396 INFO:teuthology.orchestra.run.smithi102.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-04-25T19:41:26.605 INFO:teuthology.orchestra.run.smithi102.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.4 MB/s | 309 kB 00:00 2024-04-25T19:41:26.704 INFO:teuthology.orchestra.run.smithi029.stdout:(5/32): librgw2-19.0.0-3291.g994e2f20.el9.x86_6 4.1 MB/s | 5.2 MB 00:01 2024-04-25T19:41:26.746 INFO:teuthology.orchestra.run.smithi029.stdout:(6/32): python3-ceph-argparse-19.0.0-3291.g994e 1.1 MB/s | 45 kB 00:00 2024-04-25T19:41:26.822 INFO:teuthology.orchestra.run.smithi102.stdout:(22/32): ceph-radosgw-19.0.0-3291.g994e2f20.el9 1.6 MB/s | 10 MB 00:06 2024-04-25T19:41:26.889 INFO:teuthology.orchestra.run.smithi102.stdout:(23/32): libarrow-9.0.0-10.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-04-25T19:41:26.964 INFO:teuthology.orchestra.run.smithi102.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 649 kB/s | 49 kB 00:00 2024-04-25T19:41:26.973 INFO:teuthology.orchestra.run.smithi029.stdout:(7/32): ceph-radosgw-19.0.0-3291.g994e2f20.el9. 1.6 MB/s | 10 MB 00:06 2024-04-25T19:41:26.989 INFO:teuthology.orchestra.run.smithi102.stdout:(25/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 155 kB/s | 26 kB 00:00 2024-04-25T19:41:26.999 INFO:teuthology.orchestra.run.smithi029.stdout:(8/32): python3-ceph-common-19.0.0-3291.g994e2f 535 kB/s | 135 kB 00:00 2024-04-25T19:41:27.032 INFO:teuthology.orchestra.run.smithi102.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 998 kB/s | 67 kB 00:00 2024-04-25T19:41:27.099 INFO:teuthology.orchestra.run.smithi102.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-04-25T19:41:27.199 INFO:teuthology.orchestra.run.smithi102.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-25T19:41:27.233 INFO:teuthology.orchestra.run.smithi102.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.4 MB/s | 838 kB 00:00 2024-04-25T19:41:27.316 INFO:teuthology.orchestra.run.smithi029.stdout:(9/32): ceph-common-19.0.0-3291.g994e2f20.el9.x 2.9 MB/s | 20 MB 00:06 2024-04-25T19:41:27.350 INFO:teuthology.orchestra.run.smithi029.stdout:(10/32): python3-cephfs-19.0.0-3291.g994e2f20.e 429 kB/s | 161 kB 00:00 2024-04-25T19:41:27.375 INFO:teuthology.orchestra.run.smithi029.stdout:(11/32): python3-rados-19.0.0-3291.g994e2f20.el 853 kB/s | 321 kB 00:00 2024-04-25T19:41:27.442 INFO:teuthology.orchestra.run.smithi029.stdout:(12/32): python3-rbd-19.0.0-3291.g994e2f20.el9. 2.3 MB/s | 298 kB 00:00 2024-04-25T19:41:27.468 INFO:teuthology.orchestra.run.smithi029.stdout:(13/32): python3-rgw-19.0.0-3291.g994e2f20.el9. 845 kB/s | 100 kB 00:00 2024-04-25T19:41:27.518 INFO:teuthology.orchestra.run.smithi029.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 749 kB/s | 107 kB 00:00 2024-04-25T19:41:27.541 INFO:teuthology.orchestra.run.smithi102.stdout:(30/32): ceph-common-19.0.0-3291.g994e2f20.el9. 2.8 MB/s | 20 MB 00:07 2024-04-25T19:41:27.553 INFO:teuthology.orchestra.run.smithi029.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-04-25T19:41:27.578 INFO:teuthology.orchestra.run.smithi029.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 413 kB/s | 45 kB 00:00 2024-04-25T19:41:27.620 INFO:teuthology.orchestra.run.smithi029.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 662 kB 00:00 2024-04-25T19:41:27.654 INFO:teuthology.orchestra.run.smithi029.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-04-25T19:41:27.688 INFO:teuthology.orchestra.run.smithi029.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 387 kB/s | 42 kB 00:00 2024-04-25T19:41:27.722 INFO:teuthology.orchestra.run.smithi029.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-04-25T19:41:27.756 INFO:teuthology.orchestra.run.smithi029.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 357 kB/s | 36 kB 00:00 2024-04-25T19:41:27.789 INFO:teuthology.orchestra.run.smithi029.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 705 kB/s | 47 kB 00:00 2024-04-25T19:41:27.886 INFO:teuthology.orchestra.run.smithi102.stdout:(31/32): librados2-19.0.0-3291.g994e2f20.el9.x8 4.9 MB/s | 3.3 MB 00:00 2024-04-25T19:41:28.026 INFO:teuthology.orchestra.run.smithi029.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 143 kB/s | 26 kB 00:00 2024-04-25T19:41:28.051 INFO:teuthology.orchestra.run.smithi029.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 939 kB/s | 309 kB 00:00 2024-04-25T19:41:28.077 INFO:teuthology.orchestra.run.smithi029.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 965 kB/s | 49 kB 00:00 2024-04-25T19:41:28.111 INFO:teuthology.orchestra.run.smithi029.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-04-25T19:41:28.300 INFO:teuthology.orchestra.run.smithi102.stdout:(32/32): librbd1-19.0.0-3291.g994e2f20.el9.x86_ 2.9 MB/s | 3.1 MB 00:01 2024-04-25T19:41:28.303 INFO:teuthology.orchestra.run.smithi029.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.6 MB/s | 838 kB 00:00 2024-04-25T19:41:28.303 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:41:28.304 INFO:teuthology.orchestra.run.smithi102.stdout:Total 7.1 MB/s | 58 MB 00:08 2024-04-25T19:41:28.353 INFO:teuthology.orchestra.run.smithi029.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 792 kB/s | 192 kB 00:00 2024-04-25T19:41:28.405 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-04-25T19:41:28.441 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-04-25T19:41:28.441 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-04-25T19:41:28.604 INFO:teuthology.orchestra.run.smithi029.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 5.3 MB/s | 1.6 MB 00:00 2024-04-25T19:41:28.891 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-04-25T19:41:28.892 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-04-25T19:41:28.896 INFO:teuthology.orchestra.run.smithi029.stdout:(30/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.9 MB/s | 4.4 MB 00:01 2024-04-25T19:41:29.155 INFO:teuthology.orchestra.run.smithi029.stdout:(31/32): librados2-19.0.0-3291.g994e2f20.el9.x8 4.1 MB/s | 3.3 MB 00:00 2024-04-25T19:41:29.330 INFO:teuthology.orchestra.run.smithi029.stdout:(32/32): librbd1-19.0.0-3291.g994e2f20.el9.x86_ 4.3 MB/s | 3.1 MB 00:00 2024-04-25T19:41:29.337 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:41:29.337 INFO:teuthology.orchestra.run.smithi029.stdout:Total 6.3 MB/s | 58 MB 00:09 2024-04-25T19:41:29.439 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-25T19:41:29.480 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-25T19:41:29.480 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-25T19:41:29.958 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-25T19:41:29.959 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-25T19:41:30.392 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-04-25T19:41:30.467 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-25T19:41:30.496 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-04-25T19:41:30.522 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-04-25T19:41:30.545 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-25T19:41:30.567 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-25T19:41:30.761 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-25T19:41:30.816 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-25T19:41:31.154 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-04-25T19:41:31.308 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 7/34 2024-04-25T19:41:31.391 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 7/34 2024-04-25T19:41:31.453 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-25T19:41:31.485 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 8/34 2024-04-25T19:41:31.517 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/34 2024-04-25T19:41:31.551 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-25T19:41:31.579 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/34 2024-04-25T19:41:31.589 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-04-25T19:41:31.608 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-04-25T19:41:31.615 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-04-25T19:41:31.638 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-25T19:41:31.652 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9. 11/34 2024-04-25T19:41:31.663 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-25T19:41:31.732 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 12/34 2024-04-25T19:41:31.761 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_6 13/34 2024-04-25T19:41:31.825 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_6 13/34 2024-04-25T19:41:31.859 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-04-25T19:41:31.874 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-25T19:41:31.947 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-25T19:41:31.970 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-04-25T19:41:32.002 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-04-25T19:41:32.044 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-04-25T19:41:32.069 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-04-25T19:41:32.098 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-04-25T19:41:32.130 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-04-25T19:41:32.349 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-04-25T19:41:32.456 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-04-25T19:41:32.502 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 22/34 2024-04-25T19:41:32.522 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 7/34 2024-04-25T19:41:32.591 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 22/34 2024-04-25T19:41:32.622 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 7/34 2024-04-25T19:41:32.623 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 23/34 2024-04-25T19:41:32.659 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-25T19:41:32.717 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 8/34 2024-04-25T19:41:32.741 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/34 2024-04-25T19:41:32.806 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/34 2024-04-25T19:41:32.842 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-04-25T19:41:32.896 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9. 11/34 2024-04-25T19:41:32.976 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 12/34 2024-04-25T19:41:33.005 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_6 13/34 2024-04-25T19:41:33.053 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_6 13/34 2024-04-25T19:41:33.088 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-04-25T19:41:33.151 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-25T19:41:33.179 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 26/34 2024-04-25T19:41:33.213 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-04-25T19:41:33.248 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-04-25T19:41:33.250 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 26/34 2024-04-25T19:41:33.289 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-04-25T19:41:33.320 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 27/34 2024-04-25T19:41:33.331 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-04-25T19:41:33.357 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x8 28/34 2024-04-25T19:41:33.371 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-04-25T19:41:33.412 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-04-25T19:41:33.751 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-04-25T19:41:33.809 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 22/34 2024-04-25T19:41:33.901 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 22/34 2024-04-25T19:41:33.932 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 23/34 2024-04-25T19:41:33.969 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-25T19:41:34.493 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-25T19:41:34.524 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 26/34 2024-04-25T19:41:34.604 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 26/34 2024-04-25T19:41:34.684 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 27/34 2024-04-25T19:41:34.719 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x8 28/34 2024-04-25T19:41:35.529 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 29/34 2024-04-25T19:41:35.562 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 29/34 2024-04-25T19:41:36.119 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 29/34 2024-04-25T19:41:36.157 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 30/34 2024-04-25T19:41:36.241 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 30/34 2024-04-25T19:41:36.241 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T19:41:36.241 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T19:41:36.241 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:41:36.275 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 31/34 2024-04-25T19:41:37.056 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 29/34 2024-04-25T19:41:37.098 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 29/34 2024-04-25T19:41:37.687 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 29/34 2024-04-25T19:41:37.731 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 30/34 2024-04-25T19:41:37.832 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 30/34 2024-04-25T19:41:37.832 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T19:41:37.832 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T19:41:37.832 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:41:37.867 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 31/34 2024-04-25T19:41:45.773 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 31/34 2024-04-25T19:41:45.773 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /sys 2024-04-25T19:41:45.774 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /proc 2024-04-25T19:41:45.774 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /mnt 2024-04-25T19:41:45.774 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /var/tmp 2024-04-25T19:41:45.774 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /home 2024-04-25T19:41:45.774 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /root 2024-04-25T19:41:45.774 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /tmp 2024-04-25T19:41:45.774 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:41:46.772 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 32/34 2024-04-25T19:41:46.796 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 32/34 2024-04-25T19:41:46.796 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:41:46.796 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T19:41:46.796 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T19:41:46.796 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T19:41:46.797 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:41:46.801 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-25T19:41:46.887 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-25T19:41:46.887 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-25T19:41:48.165 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-25T19:41:48.166 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 1/34 2024-04-25T19:41:48.166 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2/34 2024-04-25T19:41:48.166 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 3/34 2024-04-25T19:41:48.166 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 4/34 2024-04-25T19:41:48.166 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 5/34 2024-04-25T19:41:48.166 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_6 6/34 2024-04-25T19:41:48.166 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 7/34 2024-04-25T19:41:48.166 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9. 8/34 2024-04-25T19:41:48.166 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x8 9/34 2024-04-25T19:41:48.166 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 10/34 2024-04-25T19:41:48.166 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 11/34 2024-04-25T19:41:48.166 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 12/34 2024-04-25T19:41:48.166 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 13/34 2024-04-25T19:41:48.167 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-25T19:41:48.167 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-25T19:41:48.167 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-25T19:41:48.167 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-25T19:41:48.167 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-25T19:41:48.167 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-25T19:41:48.167 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-25T19:41:48.167 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-25T19:41:48.167 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-25T19:41:48.167 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-25T19:41:48.167 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-25T19:41:48.167 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-25T19:41:48.167 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-25T19:41:48.167 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-25T19:41:48.168 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-25T19:41:48.168 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-25T19:41:48.168 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-25T19:41:48.169 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 31/34 2024-04-25T19:41:48.169 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-25T19:41:48.169 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 33/34 2024-04-25T19:41:48.899 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-25T19:41:48.899 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:41:48.899 INFO:teuthology.orchestra.run.smithi102.stdout:Upgraded: 2024-04-25T19:41:48.899 INFO:teuthology.orchestra.run.smithi102.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:48.900 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:48.900 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-04-25T19:41:48.900 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T19:41:48.900 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:48.900 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:48.900 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:48.900 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:48.900 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T19:41:48.900 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T19:41:48.900 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T19:41:48.900 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:48.901 INFO:teuthology.orchestra.run.smithi102.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T19:41:48.901 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T19:41:48.901 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T19:41:48.901 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:48.901 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T19:41:48.901 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:48.901 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T19:41:48.901 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T19:41:48.901 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T19:41:48.901 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:48.901 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:48.901 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:48.902 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T19:41:48.902 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:48.902 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:48.902 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:48.902 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T19:41:48.902 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T19:41:48.902 INFO:teuthology.orchestra.run.smithi102.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T19:41:48.902 INFO:teuthology.orchestra.run.smithi102.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T19:41:48.902 INFO:teuthology.orchestra.run.smithi102.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T19:41:48.902 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:41:48.902 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:41:49.161 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-test 2024-04-25T19:41:49.699 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:32 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:41:49.823 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:41:49.824 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:41:49.824 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-04-25T19:41:49.824 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:41:49.824 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-04-25T19:41:49.824 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T19:41:49.825 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-04-25T19:41:49.825 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T19:41:49.825 INFO:teuthology.orchestra.run.smithi102.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T19:41:49.825 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T19:41:49.825 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:41:49.825 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-04-25T19:41:49.825 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:41:49.826 INFO:teuthology.orchestra.run.smithi102.stdout:Install 4 Packages 2024-04-25T19:41:49.826 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:41:49.826 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 44 M 2024-04-25T19:41:49.826 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 210 M 2024-04-25T19:41:49.826 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-04-25T19:41:49.984 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 31/34 2024-04-25T19:41:49.984 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /sys 2024-04-25T19:41:49.984 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /proc 2024-04-25T19:41:49.984 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /mnt 2024-04-25T19:41:49.984 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /var/tmp 2024-04-25T19:41:49.984 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /home 2024-04-25T19:41:49.984 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /root 2024-04-25T19:41:49.984 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /tmp 2024-04-25T19:41:49.984 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:41:50.154 INFO:teuthology.orchestra.run.smithi102.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 305 kB 00:00 2024-04-25T19:41:50.229 INFO:teuthology.orchestra.run.smithi102.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 843 kB/s | 64 kB 00:00 2024-04-25T19:41:50.305 INFO:teuthology.orchestra.run.smithi102.stdout:(3/4): libcephsqlite-19.0.0-3291.g994e2f20.el9. 416 kB/s | 158 kB 00:00 2024-04-25T19:41:51.016 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 32/34 2024-04-25T19:41:51.041 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 32/34 2024-04-25T19:41:51.042 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:41:51.042 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T19:41:51.042 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T19:41:51.042 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T19:41:51.042 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:41:51.044 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-25T19:41:51.130 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-25T19:41:51.131 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-25T19:41:52.416 INFO:teuthology.orchestra.run.smithi102.stdout:(4/4): ceph-test-19.0.0-3291.g994e2f20.el9.x86_ 17 MB/s | 43 MB 00:02 2024-04-25T19:41:52.419 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:41:52.419 INFO:teuthology.orchestra.run.smithi102.stdout:Total 17 MB/s | 44 MB 00:02 2024-04-25T19:41:52.444 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-04-25T19:41:52.464 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-04-25T19:41:52.464 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-04-25T19:41:52.798 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-04-25T19:41:52.799 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-04-25T19:41:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-25T19:41:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 1/34 2024-04-25T19:41:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2/34 2024-04-25T19:41:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 3/34 2024-04-25T19:41:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 4/34 2024-04-25T19:41:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 5/34 2024-04-25T19:41:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_6 6/34 2024-04-25T19:41:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 7/34 2024-04-25T19:41:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9. 8/34 2024-04-25T19:41:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x8 9/34 2024-04-25T19:41:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 10/34 2024-04-25T19:41:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 11/34 2024-04-25T19:41:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 12/34 2024-04-25T19:41:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 13/34 2024-04-25T19:41:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-25T19:41:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-25T19:41:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-25T19:41:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-25T19:41:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-25T19:41:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-25T19:41:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-25T19:41:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-25T19:41:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-25T19:41:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-25T19:41:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-25T19:41:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-25T19:41:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-25T19:41:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-25T19:41:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-25T19:41:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-25T19:41:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-25T19:41:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 31/34 2024-04-25T19:41:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-25T19:41:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 33/34 2024-04-25T19:41:53.279 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-04-25T19:41:53.349 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-25T19:41:53.390 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-25T19:41:53.418 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 3/4 2024-04-25T19:41:53.720 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-25T19:41:53.721 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:41:53.721 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2024-04-25T19:41:53.721 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:53.721 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:53.721 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-25T19:41:53.721 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T19:41:53.721 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:53.721 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:53.721 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:53.721 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:53.721 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T19:41:53.721 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T19:41:53.721 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T19:41:53.722 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:53.722 INFO:teuthology.orchestra.run.smithi029.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T19:41:53.722 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T19:41:53.722 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T19:41:53.722 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:53.722 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T19:41:53.722 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:53.722 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T19:41:53.722 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T19:41:53.722 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T19:41:53.723 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:53.723 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:53.723 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:53.723 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T19:41:53.723 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:53.723 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:53.723 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:41:53.723 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T19:41:53.723 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T19:41:53.723 INFO:teuthology.orchestra.run.smithi029.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T19:41:53.723 INFO:teuthology.orchestra.run.smithi029.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T19:41:53.723 INFO:teuthology.orchestra.run.smithi029.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T19:41:53.724 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:41:53.724 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:41:54.071 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-test 2024-04-25T19:41:54.640 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:37 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:41:54.764 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:41:54.765 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:41:54.765 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-25T19:41:54.765 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:41:54.765 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-25T19:41:54.765 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T19:41:54.765 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-25T19:41:54.765 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T19:41:54.765 INFO:teuthology.orchestra.run.smithi029.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T19:41:54.765 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T19:41:54.766 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:41:54.766 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-25T19:41:54.766 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:41:54.766 INFO:teuthology.orchestra.run.smithi029.stdout:Install 4 Packages 2024-04-25T19:41:54.766 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:41:54.767 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 44 M 2024-04-25T19:41:54.767 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 210 M 2024-04-25T19:41:54.767 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-25T19:41:55.262 INFO:teuthology.orchestra.run.smithi029.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 305 kB 00:00 2024-04-25T19:41:55.322 INFO:teuthology.orchestra.run.smithi029.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 64 kB 00:00 2024-04-25T19:41:55.364 INFO:teuthology.orchestra.run.smithi029.stdout:(3/4): libcephsqlite-19.0.0-3291.g994e2f20.el9. 473 kB/s | 158 kB 00:00 2024-04-25T19:41:57.506 INFO:teuthology.orchestra.run.smithi029.stdout:(4/4): ceph-test-19.0.0-3291.g994e2f20.el9.x86_ 17 MB/s | 43 MB 00:02 2024-04-25T19:41:57.509 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:41:57.509 INFO:teuthology.orchestra.run.smithi029.stdout:Total 16 MB/s | 44 MB 00:02 2024-04-25T19:41:57.536 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-25T19:41:57.559 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-25T19:41:57.559 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-25T19:41:57.673 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 3/4 2024-04-25T19:41:57.748 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 4/4 2024-04-25T19:41:57.914 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-25T19:41:57.914 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-25T19:41:58.417 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-25T19:41:58.509 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-25T19:41:58.565 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-25T19:41:58.601 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 3/4 2024-04-25T19:42:00.151 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 4/4 2024-04-25T19:42:00.151 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 1/4 2024-04-25T19:42:00.151 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2/4 2024-04-25T19:42:00.151 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-25T19:42:00.411 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-25T19:42:00.411 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:00.411 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-04-25T19:42:00.411 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:42:00.411 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:42:00.411 INFO:teuthology.orchestra.run.smithi102.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T19:42:00.411 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T19:42:00.412 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:00.412 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:42:00.646 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph 2024-04-25T19:42:01.164 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:44 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:01.294 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:42:01.296 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:01.297 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-04-25T19:42:01.297 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:01.297 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-04-25T19:42:01.297 INFO:teuthology.orchestra.run.smithi102.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T19:42:01.297 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-04-25T19:42:01.297 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T19:42:01.297 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T19:42:01.297 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T19:42:01.297 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T19:42:01.297 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T19:42:01.297 INFO:teuthology.orchestra.run.smithi102.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T19:42:01.297 INFO:teuthology.orchestra.run.smithi102.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T19:42:01.297 INFO:teuthology.orchestra.run.smithi102.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T19:42:01.298 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T19:42:01.298 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T19:42:01.298 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T19:42:01.298 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T19:42:01.298 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T19:42:01.298 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T19:42:01.298 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T19:42:01.298 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections 2024-04-25T19:42:01.298 INFO:teuthology.orchestra.run.smithi102.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-25T19:42:01.298 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T19:42:01.298 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T19:42:01.298 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T19:42:01.298 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T19:42:01.298 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T19:42:01.299 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T19:42:01.299 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T19:42:01.299 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T19:42:01.299 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T19:42:01.299 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T19:42:01.299 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T19:42:01.299 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T19:42:01.299 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T19:42:01.299 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T19:42:01.299 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T19:42:01.299 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T19:42:01.299 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T19:42:01.299 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T19:42:01.299 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T19:42:01.299 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:01.300 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-04-25T19:42:01.300 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:01.300 INFO:teuthology.orchestra.run.smithi102.stdout:Install 36 Packages 2024-04-25T19:42:01.300 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:01.300 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 31 M 2024-04-25T19:42:01.300 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 117 M 2024-04-25T19:42:01.300 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-04-25T19:42:01.910 INFO:teuthology.orchestra.run.smithi102.stdout:(1/36): ceph-19.0.0-3291.g994e2f20.el9.x86_64.r 32 kB/s | 6.6 kB 00:00 2024-04-25T19:42:02.060 INFO:teuthology.orchestra.run.smithi102.stdout:(2/36): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86_ 4.2 MB/s | 1.5 MB 00:00 2024-04-25T19:42:02.978 INFO:teuthology.orchestra.run.smithi102.stdout:(3/36): ceph-osd-19.0.0-3291.g994e2f20.el9.x86_ 19 MB/s | 18 MB 00:00 2024-04-25T19:42:03.028 INFO:teuthology.orchestra.run.smithi102.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3291.g994e 4.9 MB/s | 249 kB 00:00 2024-04-25T19:42:03.132 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 3/4 2024-04-25T19:42:03.232 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 4/4 2024-04-25T19:42:03.253 INFO:teuthology.orchestra.run.smithi102.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-04-25T19:42:03.395 INFO:teuthology.orchestra.run.smithi102.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 8.7 MB/s | 1.2 MB 00:00 2024-04-25T19:42:03.428 INFO:teuthology.orchestra.run.smithi102.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 3.1 MB/s | 106 kB 00:00 2024-04-25T19:42:03.470 INFO:teuthology.orchestra.run.smithi102.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-04-25T19:42:03.512 INFO:teuthology.orchestra.run.smithi102.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-04-25T19:42:03.546 INFO:teuthology.orchestra.run.smithi102.stdout:(10/36): python3-urllib3-1.26.5-5.el9.noarch.rp 6.3 MB/s | 215 kB 00:00 2024-04-25T19:42:03.580 INFO:teuthology.orchestra.run.smithi102.stdout:(11/36): lua-5.4.4-4.el9.x86_64.rpm 5.5 MB/s | 188 kB 00:00 2024-04-25T19:42:03.613 INFO:teuthology.orchestra.run.smithi102.stdout:(12/36): python3-mako-1.1.4-6.el9.noarch.rpm 5.1 MB/s | 172 kB 00:00 2024-04-25T19:42:03.647 INFO:teuthology.orchestra.run.smithi102.stdout:(13/36): python3-markupsafe-1.1.1-12.el9.x86_64 1.0 MB/s | 35 kB 00:00 2024-04-25T19:42:03.680 INFO:teuthology.orchestra.run.smithi102.stdout:(14/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.5 MB/s | 51 kB 00:00 2024-04-25T19:42:03.714 INFO:teuthology.orchestra.run.smithi102.stdout:(15/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.2 MB/s | 42 kB 00:00 2024-04-25T19:42:03.748 INFO:teuthology.orchestra.run.smithi102.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 669 kB/s | 22 kB 00:00 2024-04-25T19:42:03.915 INFO:teuthology.orchestra.run.smithi102.stdout:(17/36): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 1.0 MB/s | 2.3 MB 00:02 2024-04-25T19:42:04.190 INFO:teuthology.orchestra.run.smithi102.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 159 kB/s | 43 kB 00:00 2024-04-25T19:42:04.216 INFO:teuthology.orchestra.run.smithi102.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 323 kB/s | 151 kB 00:00 2024-04-25T19:42:04.299 INFO:teuthology.orchestra.run.smithi102.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-04-25T19:42:04.366 INFO:teuthology.orchestra.run.smithi102.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.3 MB/s | 358 kB 00:00 2024-04-25T19:42:04.391 INFO:teuthology.orchestra.run.smithi102.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-04-25T19:42:04.433 INFO:teuthology.orchestra.run.smithi102.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 265 kB/s | 18 kB 00:00 2024-04-25T19:42:04.438 INFO:teuthology.orchestra.run.smithi102.stdout:[MIRROR] python3-jaraco-collections-3.0.0-8.el9.noarch.rpm: Status code: 404 for https://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-jaraco-collections-3.0.0-8.el9.noarch.rpm (IP: 173.208.208.139) 2024-04-25T19:42:04.480 INFO:teuthology.orchestra.run.smithi102.stdout:[MIRROR] python3-jaraco-functools-3.5.0-2.el9.noarch.rpm: Status code: 404 for https://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-jaraco-functools-3.5.0-2.el9.noarch.rpm (IP: 173.208.208.139) 2024-04-25T19:42:04.526 INFO:teuthology.orchestra.run.smithi102.stdout:[MIRROR] python3-jaraco-collections-3.0.0-8.el9.noarch.rpm: Status code: 404 for http://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-jaraco-collections-3.0.0-8.el9.noarch.rpm (IP: 173.208.208.139) 2024-04-25T19:42:04.568 INFO:teuthology.orchestra.run.smithi102.stdout:[MIRROR] python3-jaraco-functools-3.5.0-2.el9.noarch.rpm: Status code: 404 for http://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-jaraco-functools-3.5.0-2.el9.noarch.rpm (IP: 173.208.208.139) 2024-04-25T19:42:04.625 INFO:teuthology.orchestra.run.smithi102.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 99 kB/s | 23 kB 00:00 2024-04-25T19:42:04.667 INFO:teuthology.orchestra.run.smithi102.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 83 kB/s | 19 kB 00:00 2024-04-25T19:42:04.817 INFO:teuthology.orchestra.run.smithi102.stdout:(26/36): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 1.7 MB/s | 4.9 MB 00:02 2024-04-25T19:42:04.916 INFO:teuthology.orchestra.run.smithi102.stdout:[MIRROR] python3-jaraco-text-3.2.0-6.el9.noarch.rpm: Status code: 404 for http://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-jaraco-text-3.2.0-6.el9.noarch.rpm (IP: 173.208.208.139) 2024-04-25T19:42:04.956 INFO:teuthology.orchestra.run.smithi102.stdout:[MIRROR] python3-logutils-0.3.5-21.el9.noarch.rpm: Status code: 404 for http://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-logutils-0.3.5-21.el9.noarch.rpm (IP: 173.208.208.139) 2024-04-25T19:42:04.962 INFO:teuthology.orchestra.run.smithi102.stdout:[MIRROR] python3-jaraco-text-3.2.0-6.el9.noarch.rpm: Status code: 404 for https://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-jaraco-text-3.2.0-6.el9.noarch.rpm (IP: 173.208.208.139) 2024-04-25T19:42:05.018 INFO:teuthology.orchestra.run.smithi102.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 50 kB/s | 20 kB 00:00 2024-04-25T19:42:05.018 INFO:teuthology.orchestra.run.smithi102.stdout:[MIRROR] python3-logutils-0.3.5-21.el9.noarch.rpm: Status code: 404 for https://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-logutils-0.3.5-21.el9.noarch.rpm (IP: 173.208.208.139) 2024-04-25T19:42:05.064 INFO:teuthology.orchestra.run.smithi102.stdout:[MIRROR] python3-pecan-1.4.2-3.el9.noarch.rpm: Status code: 404 for https://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-pecan-1.4.2-3.el9.noarch.rpm (IP: 173.208.208.139) 2024-04-25T19:42:05.101 INFO:teuthology.orchestra.run.smithi102.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 107 kB/s | 46 kB 00:00 2024-04-25T19:42:05.152 INFO:teuthology.orchestra.run.smithi102.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 329 kB/s | 16 kB 00:00 2024-04-25T19:42:05.174 INFO:teuthology.orchestra.run.smithi102.stdout:[MIRROR] python3-more-itertools-8.12.0-2.el9.noarch.rpm: Status code: 404 for http://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-more-itertools-8.12.0-2.el9.noarch.rpm (IP: 173.208.208.139) 2024-04-25T19:42:05.210 INFO:teuthology.orchestra.run.smithi102.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-04-25T19:42:05.235 INFO:teuthology.orchestra.run.smithi102.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-04-25T19:42:05.261 INFO:teuthology.orchestra.run.smithi102.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-04-25T19:42:05.319 INFO:teuthology.orchestra.run.smithi102.stdout:(33/36): python3-more-itertools-8.12.0-2.el9.no 157 kB/s | 79 kB 00:00 2024-04-25T19:42:05.345 INFO:teuthology.orchestra.run.smithi102.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-25T19:42:05.370 INFO:teuthology.orchestra.run.smithi102.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-04-25T19:42:05.395 INFO:teuthology.orchestra.run.smithi102.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 265 kB/s | 20 kB 00:00 2024-04-25T19:42:05.399 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:42:05.399 INFO:teuthology.orchestra.run.smithi102.stdout:Total 7.6 MB/s | 31 MB 00:04 2024-04-25T19:42:05.538 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-04-25T19:42:05.577 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-04-25T19:42:05.577 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-04-25T19:42:05.872 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-04-25T19:42:05.874 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-04-25T19:42:07.129 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-04-25T19:42:07.189 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 4/4 2024-04-25T19:42:07.189 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 1/4 2024-04-25T19:42:07.189 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2/4 2024-04-25T19:42:07.189 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-25T19:42:07.190 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-25T19:42:07.222 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-25T19:42:07.250 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-25T19:42:07.324 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-25T19:42:07.383 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-25T19:42:07.416 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-25T19:42:07.485 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-25T19:42:07.485 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:07.485 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-25T19:42:07.485 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:42:07.485 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:42:07.485 INFO:teuthology.orchestra.run.smithi029.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T19:42:07.485 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T19:42:07.485 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:07.485 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:42:07.509 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-25T19:42:07.587 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-25T19:42:07.635 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-25T19:42:07.670 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-25T19:42:07.700 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-25T19:42:07.726 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-25T19:42:07.759 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-25T19:42:07.800 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph 2024-04-25T19:42:07.801 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-25T19:42:07.852 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-25T19:42:07.886 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-25T19:42:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-25T19:42:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-25T19:42:08.147 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-25T19:42:08.225 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-25T19:42:08.278 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-25T19:42:08.324 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-25T19:42:08.334 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:51 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:08.381 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-25T19:42:08.438 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-25T19:42:08.463 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:42:08.465 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:42:08.465 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-25T19:42:08.465 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:42:08.465 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-25T19:42:08.466 INFO:teuthology.orchestra.run.smithi029.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T19:42:08.466 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-25T19:42:08.466 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T19:42:08.466 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T19:42:08.466 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T19:42:08.466 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T19:42:08.466 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T19:42:08.466 INFO:teuthology.orchestra.run.smithi029.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T19:42:08.466 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T19:42:08.466 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T19:42:08.466 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T19:42:08.466 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T19:42:08.466 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T19:42:08.467 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T19:42:08.467 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T19:42:08.467 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T19:42:08.467 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T19:42:08.467 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections 2024-04-25T19:42:08.467 INFO:teuthology.orchestra.run.smithi029.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-25T19:42:08.467 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T19:42:08.467 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T19:42:08.467 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T19:42:08.467 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T19:42:08.467 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T19:42:08.467 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T19:42:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T19:42:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T19:42:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T19:42:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T19:42:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T19:42:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T19:42:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T19:42:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T19:42:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T19:42:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T19:42:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T19:42:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T19:42:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T19:42:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:08.468 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-25T19:42:08.468 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:42:08.469 INFO:teuthology.orchestra.run.smithi029.stdout:Install 36 Packages 2024-04-25T19:42:08.469 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:08.469 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 31 M 2024-04-25T19:42:08.469 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 117 M 2024-04-25T19:42:08.469 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-25T19:42:08.626 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-25T19:42:08.678 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-25T19:42:08.757 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-25T19:42:08.855 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-25T19:42:08.894 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-25T19:42:09.013 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-25T19:42:09.179 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9. 31/36 2024-04-25T19:42:09.204 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 32/36 2024-04-25T19:42:09.229 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 32/36 2024-04-25T19:42:09.230 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:42:09.230 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T19:42:09.230 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T19:42:09.230 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T19:42:09.230 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:09.406 INFO:teuthology.orchestra.run.smithi029.stdout:(1/36): ceph-19.0.0-3291.g994e2f20.el9.x86_64.r 30 kB/s | 6.6 kB 00:00 2024-04-25T19:42:09.599 INFO:teuthology.orchestra.run.smithi029.stdout:(2/36): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86_ 3.6 MB/s | 1.5 MB 00:00 2024-04-25T19:42:09.683 INFO:teuthology.orchestra.run.smithi029.stdout:(3/36): ceph-mds-19.0.0-3291.g994e2f20.el9.x86_ 4.6 MB/s | 2.3 MB 00:00 2024-04-25T19:42:09.758 INFO:teuthology.orchestra.run.smithi029.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3291.g994e 3.2 MB/s | 249 kB 00:00 2024-04-25T19:42:09.884 INFO:teuthology.orchestra.run.smithi029.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.0 MB/s | 253 kB 00:00 2024-04-25T19:42:09.961 INFO:teuthology.orchestra.run.smithi029.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 16 MB/s | 1.2 MB 00:00 2024-04-25T19:42:09.994 INFO:teuthology.orchestra.run.smithi029.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 3.1 MB/s | 106 kB 00:00 2024-04-25T19:42:10.028 INFO:teuthology.orchestra.run.smithi029.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 3.9 MB/s | 135 kB 00:00 2024-04-25T19:42:10.070 INFO:teuthology.orchestra.run.smithi029.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-04-25T19:42:10.104 INFO:teuthology.orchestra.run.smithi029.stdout:(10/36): python3-urllib3-1.26.5-5.el9.noarch.rp 6.3 MB/s | 215 kB 00:00 2024-04-25T19:42:10.146 INFO:teuthology.orchestra.run.smithi029.stdout:(11/36): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 188 kB 00:00 2024-04-25T19:42:10.180 INFO:teuthology.orchestra.run.smithi029.stdout:(12/36): python3-mako-1.1.4-6.el9.noarch.rpm 5.0 MB/s | 172 kB 00:00 2024-04-25T19:42:10.214 INFO:teuthology.orchestra.run.smithi029.stdout:(13/36): python3-markupsafe-1.1.1-12.el9.x86_64 1.0 MB/s | 35 kB 00:00 2024-04-25T19:42:10.248 INFO:teuthology.orchestra.run.smithi029.stdout:(14/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.5 MB/s | 51 kB 00:00 2024-04-25T19:42:10.282 INFO:teuthology.orchestra.run.smithi029.stdout:(15/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.2 MB/s | 42 kB 00:00 2024-04-25T19:42:10.316 INFO:teuthology.orchestra.run.smithi029.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 665 kB/s | 22 kB 00:00 2024-04-25T19:42:10.652 INFO:teuthology.orchestra.run.smithi029.stdout:(17/36): ceph-osd-19.0.0-3291.g994e2f20.el9.x86 17 MB/s | 18 MB 00:01 2024-04-25T19:42:10.742 INFO:teuthology.orchestra.run.smithi029.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 490 kB/s | 43 kB 00:00 2024-04-25T19:42:10.767 INFO:teuthology.orchestra.run.smithi029.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 335 kB/s | 151 kB 00:00 2024-04-25T19:42:10.772 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 33/36 2024-04-25T19:42:10.799 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 33/36 2024-04-25T19:42:10.799 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:42:10.799 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T19:42:10.799 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T19:42:10.799 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T19:42:10.799 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:10.809 INFO:teuthology.orchestra.run.smithi029.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-04-25T19:42:10.852 INFO:teuthology.orchestra.run.smithi029.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 256 kB/s | 11 kB 00:00 2024-04-25T19:42:10.879 INFO:teuthology.orchestra.run.smithi029.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.2 MB/s | 358 kB 00:00 2024-04-25T19:42:10.904 INFO:teuthology.orchestra.run.smithi029.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 338 kB/s | 18 kB 00:00 2024-04-25T19:42:10.930 INFO:teuthology.orchestra.run.smithi029.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 456 kB/s | 23 kB 00:00 2024-04-25T19:42:10.955 INFO:teuthology.orchestra.run.smithi029.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 383 kB/s | 19 kB 00:00 2024-04-25T19:42:10.981 INFO:teuthology.orchestra.run.smithi029.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-04-25T19:42:11.006 INFO:teuthology.orchestra.run.smithi029.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 912 kB/s | 46 kB 00:00 2024-04-25T19:42:11.040 INFO:teuthology.orchestra.run.smithi029.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-04-25T19:42:11.074 INFO:teuthology.orchestra.run.smithi029.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.9 MB/s | 272 kB 00:00 2024-04-25T19:42:11.100 INFO:teuthology.orchestra.run.smithi029.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 279 kB/s | 16 kB 00:00 2024-04-25T19:42:11.126 INFO:teuthology.orchestra.run.smithi029.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-04-25T19:42:11.142 INFO:teuthology.orchestra.run.smithi029.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 846 kB/s | 36 kB 00:00 2024-04-25T19:42:11.168 INFO:teuthology.orchestra.run.smithi029.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 5.3 MB/s | 230 kB 00:00 2024-04-25T19:42:11.261 INFO:teuthology.orchestra.run.smithi029.stdout:(34/36): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 2.6 MB/s | 4.9 MB 00:01 2024-04-25T19:42:11.264 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 34/36 2024-04-25T19:42:11.286 INFO:teuthology.orchestra.run.smithi029.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 170 kB/s | 20 kB 00:00 2024-04-25T19:42:11.288 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 34/36 2024-04-25T19:42:11.288 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:42:11.288 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T19:42:11.288 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T19:42:11.288 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T19:42:11.288 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:11.320 INFO:teuthology.orchestra.run.smithi029.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2024-04-25T19:42:11.323 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:42:11.323 INFO:teuthology.orchestra.run.smithi029.stdout:Total 11 MB/s | 31 MB 00:02 2024-04-25T19:42:11.460 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-25T19:42:11.502 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-25T19:42:11.502 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-25T19:42:11.515 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 35/36 2024-04-25T19:42:11.538 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 35/36 2024-04-25T19:42:11.538 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:42:11.538 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T19:42:11.538 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T19:42:11.538 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T19:42:11.538 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:11.602 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 36/36 2024-04-25T19:42:11.821 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-25T19:42:11.821 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-25T19:42:13.095 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-25T19:42:13.166 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-25T19:42:13.209 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-25T19:42:13.245 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-25T19:42:13.329 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-25T19:42:13.379 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-25T19:42:13.413 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-25T19:42:13.425 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 36/36 2024-04-25T19:42:13.425 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/36 2024-04-25T19:42:13.425 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2/36 2024-04-25T19:42:13.425 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 3/36 2024-04-25T19:42:13.425 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 4/36 2024-04-25T19:42:13.425 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 5/36 2024-04-25T19:42:13.425 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9. 6/36 2024-04-25T19:42:13.425 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-25T19:42:13.425 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-25T19:42:13.425 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-25T19:42:13.425 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-25T19:42:13.425 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-25T19:42:13.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-25T19:42:13.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-25T19:42:13.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-25T19:42:13.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-25T19:42:13.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-25T19:42:13.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-25T19:42:13.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-25T19:42:13.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-25T19:42:13.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-25T19:42:13.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-25T19:42:13.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-25T19:42:13.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-25T19:42:13.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-25T19:42:13.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-25T19:42:13.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-25T19:42:13.427 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-25T19:42:13.427 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-25T19:42:13.428 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-25T19:42:13.428 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-25T19:42:13.428 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-25T19:42:13.428 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-25T19:42:13.428 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-25T19:42:13.428 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-25T19:42:13.428 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-25T19:42:13.511 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-25T19:42:13.597 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-25T19:42:13.644 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-25T19:42:13.680 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-25T19:42:13.718 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-25T19:42:13.753 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-25T19:42:13.787 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-25T19:42:13.830 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-25T19:42:13.874 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-25T19:42:13.917 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-25T19:42:13.955 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-25T19:42:14.042 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-25T19:42:14.203 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-25T19:42:14.273 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-25T19:42:14.273 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:14.273 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-04-25T19:42:14.274 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:42:14.274 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:42:14.274 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:42:14.274 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:42:14.274 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:42:14.274 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:42:14.274 INFO:teuthology.orchestra.run.smithi102.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T19:42:14.274 INFO:teuthology.orchestra.run.smithi102.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T19:42:14.274 INFO:teuthology.orchestra.run.smithi102.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T19:42:14.274 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T19:42:14.274 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T19:42:14.274 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T19:42:14.274 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T19:42:14.274 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T19:42:14.275 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T19:42:14.275 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T19:42:14.275 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T19:42:14.275 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T19:42:14.275 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T19:42:14.275 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T19:42:14.275 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T19:42:14.275 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T19:42:14.275 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T19:42:14.275 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T19:42:14.275 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T19:42:14.275 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T19:42:14.275 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T19:42:14.275 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T19:42:14.275 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T19:42:14.276 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T19:42:14.276 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T19:42:14.276 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T19:42:14.276 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T19:42:14.276 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T19:42:14.276 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T19:42:14.276 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T19:42:14.276 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:14.276 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:42:14.298 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-25T19:42:14.357 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-25T19:42:14.404 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-25T19:42:14.463 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-25T19:42:14.529 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-25T19:42:14.632 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-base 2024-04-25T19:42:14.736 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-25T19:42:14.796 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-25T19:42:14.877 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-25T19:42:14.977 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-25T19:42:15.015 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-25T19:42:15.147 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:58 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:15.150 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-25T19:42:15.226 INFO:teuthology.orchestra.run.smithi102.stdout:Package ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 is already installed. 2024-04-25T19:42:15.267 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:42:15.268 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-04-25T19:42:15.268 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:42:15.326 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9. 31/36 2024-04-25T19:42:15.344 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install cephadm 2024-04-25T19:42:15.352 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 32/36 2024-04-25T19:42:15.378 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 32/36 2024-04-25T19:42:15.379 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:42:15.379 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T19:42:15.379 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T19:42:15.379 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T19:42:15.379 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:15.853 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:58 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:15.972 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:42:15.973 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:15.973 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-04-25T19:42:15.973 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:15.973 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-04-25T19:42:15.973 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T19:42:15.973 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:15.973 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-04-25T19:42:15.973 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:15.973 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-04-25T19:42:15.973 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:15.973 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 762 k 2024-04-25T19:42:15.974 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 769 k 2024-04-25T19:42:15.974 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-04-25T19:42:16.273 INFO:teuthology.orchestra.run.smithi102.stdout:cephadm-19.0.0-3291.g994e2f20.el9.noarch.rpm 2.5 MB/s | 762 kB 00:00 2024-04-25T19:42:16.273 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:42:16.273 INFO:teuthology.orchestra.run.smithi102.stdout:Total 2.5 MB/s | 762 kB 00:00 2024-04-25T19:42:16.274 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-04-25T19:42:16.278 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-04-25T19:42:16.279 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-04-25T19:42:16.284 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-04-25T19:42:16.284 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-04-25T19:42:16.357 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-04-25T19:42:16.756 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 1/1 2024-04-25T19:42:16.842 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 1/1 2024-04-25T19:42:17.006 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 33/36 2024-04-25T19:42:17.034 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 33/36 2024-04-25T19:42:17.034 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:42:17.034 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T19:42:17.034 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T19:42:17.034 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T19:42:17.034 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:17.195 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 1/1 2024-04-25T19:42:17.413 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 1/1 2024-04-25T19:42:17.413 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:17.414 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-04-25T19:42:17.414 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:42:17.414 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:17.414 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:42:17.524 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 34/36 2024-04-25T19:42:17.548 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 34/36 2024-04-25T19:42:17.549 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:42:17.549 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T19:42:17.549 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T19:42:17.549 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T19:42:17.549 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:17.635 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-immutable-object-cache 2024-04-25T19:42:17.783 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 35/36 2024-04-25T19:42:17.808 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 35/36 2024-04-25T19:42:17.808 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:42:17.808 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T19:42:17.808 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T19:42:17.808 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T19:42:17.808 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:17.874 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 36/36 2024-04-25T19:42:18.147 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:01 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:18.270 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:42:18.271 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:18.271 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-04-25T19:42:18.271 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:18.271 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-04-25T19:42:18.271 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T19:42:18.271 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:18.271 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-04-25T19:42:18.271 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:18.271 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-04-25T19:42:18.271 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:18.272 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 147 k 2024-04-25T19:42:18.272 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 477 k 2024-04-25T19:42:18.272 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-04-25T19:42:18.496 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-immutable-object-cache-19.0.0-3291.g994e2f 656 kB/s | 147 kB 00:00 2024-04-25T19:42:18.496 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:42:18.497 INFO:teuthology.orchestra.run.smithi102.stdout:Total 652 kB/s | 147 kB 00:00 2024-04-25T19:42:18.497 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-04-25T19:42:18.504 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-04-25T19:42:18.504 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-04-25T19:42:18.560 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-04-25T19:42:18.560 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-04-25T19:42:18.701 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-04-25T19:42:18.780 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20. 1/1 2024-04-25T19:42:18.804 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20. 1/1 2024-04-25T19:42:18.804 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:42:18.804 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T19:42:18.804 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:19.641 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20. 1/1 2024-04-25T19:42:19.642 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:19.642 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-04-25T19:42:19.642 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:42:19.642 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:19.642 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:42:19.838 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr 2024-04-25T19:42:20.343 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:03 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:20.421 INFO:teuthology.orchestra.run.smithi102.stdout:Package ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 is already installed. 2024-04-25T19:42:20.462 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:42:20.463 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-04-25T19:42:20.463 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:42:20.543 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-dashboard 2024-04-25T19:42:20.634 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 36/36 2024-04-25T19:42:20.635 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/36 2024-04-25T19:42:20.635 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2/36 2024-04-25T19:42:20.635 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 3/36 2024-04-25T19:42:20.635 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 4/36 2024-04-25T19:42:20.635 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 5/36 2024-04-25T19:42:20.635 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9. 6/36 2024-04-25T19:42:20.635 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-25T19:42:20.635 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-25T19:42:20.635 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-25T19:42:20.635 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-25T19:42:20.635 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-25T19:42:20.635 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-25T19:42:20.635 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-25T19:42:20.636 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-25T19:42:20.636 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-25T19:42:20.636 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-25T19:42:20.636 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-25T19:42:20.636 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-25T19:42:20.636 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-25T19:42:20.636 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-25T19:42:20.636 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-25T19:42:20.636 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-25T19:42:20.636 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-25T19:42:20.636 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-25T19:42:20.636 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-25T19:42:20.636 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-25T19:42:20.636 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-25T19:42:20.637 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-25T19:42:20.637 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-25T19:42:20.637 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-25T19:42:20.637 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-25T19:42:20.637 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-25T19:42:20.637 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-25T19:42:20.637 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-25T19:42:20.637 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-25T19:42:21.052 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:04 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:21.173 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:42:21.174 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:21.174 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-04-25T19:42:21.174 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:21.174 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-04-25T19:42:21.174 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T19:42:21.174 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-04-25T19:42:21.174 INFO:teuthology.orchestra.run.smithi102.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T19:42:21.174 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T19:42:21.175 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T19:42:21.175 INFO:teuthology.orchestra.run.smithi102.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T19:42:21.175 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T19:42:21.175 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T19:42:21.175 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T19:42:21.175 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T19:42:21.175 INFO:teuthology.orchestra.run.smithi102.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T19:42:21.175 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T19:42:21.175 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T19:42:21.175 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:21.175 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-04-25T19:42:21.175 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:21.175 INFO:teuthology.orchestra.run.smithi102.stdout:Install 12 Packages 2024-04-25T19:42:21.175 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:21.176 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 8.6 M 2024-04-25T19:42:21.176 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 104 M 2024-04-25T19:42:21.176 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-04-25T19:42:21.644 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-25T19:42:21.644 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:21.645 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-25T19:42:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:42:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:42:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:42:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:42:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:42:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:42:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T19:42:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T19:42:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T19:42:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T19:42:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T19:42:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T19:42:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T19:42:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T19:42:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T19:42:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T19:42:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T19:42:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T19:42:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T19:42:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T19:42:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T19:42:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T19:42:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T19:42:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T19:42:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T19:42:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T19:42:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T19:42:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T19:42:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T19:42:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T19:42:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T19:42:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T19:42:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T19:42:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T19:42:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T19:42:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T19:42:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:21.647 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:42:21.884 INFO:teuthology.orchestra.run.smithi102.stdout:(1/12): ceph-grafana-dashboards-19.0.0-3291.g99 188 kB/s | 34 kB 00:00 2024-04-25T19:42:22.001 INFO:teuthology.orchestra.run.smithi102.stdout:(2/12): ceph-prometheus-alerts-19.0.0-3291.g994 55 kB/s | 16 kB 00:00 2024-04-25T19:42:22.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-base 2024-04-25T19:42:22.067 INFO:teuthology.orchestra.run.smithi102.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-04-25T19:42:22.093 INFO:teuthology.orchestra.run.smithi102.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.9 MB/s | 269 kB 00:00 2024-04-25T19:42:22.118 INFO:teuthology.orchestra.run.smithi102.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 17 MB/s | 863 kB 00:00 2024-04-25T19:42:22.227 INFO:teuthology.orchestra.run.smithi102.stdout:(6/12): ceph-mgr-dashboard-19.0.0-3291.g994e2f2 6.8 MB/s | 3.5 MB 00:00 2024-04-25T19:42:22.252 INFO:teuthology.orchestra.run.smithi102.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 145 kB/s | 19 kB 00:00 2024-04-25T19:42:22.419 INFO:teuthology.orchestra.run.smithi102.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 867 kB/s | 144 kB 00:00 2024-04-25T19:42:22.444 INFO:teuthology.orchestra.run.smithi102.stdout:(9/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.5 MB/s | 548 kB 00:00 2024-04-25T19:42:22.478 INFO:teuthology.orchestra.run.smithi102.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 524 kB/s | 31 kB 00:00 2024-04-25T19:42:22.511 INFO:teuthology.orchestra.run.smithi102.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-04-25T19:42:22.594 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:05 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:22.628 INFO:teuthology.orchestra.run.smithi102.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.1 MB/s | 2.0 MB 00:00 2024-04-25T19:42:22.631 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:42:22.631 INFO:teuthology.orchestra.run.smithi102.stdout:Total 5.9 MB/s | 8.6 MB 00:01 2024-04-25T19:42:22.676 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 is already installed. 2024-04-25T19:42:22.678 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-04-25T19:42:22.695 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-04-25T19:42:22.695 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-04-25T19:42:22.719 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:42:22.720 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-25T19:42:22.720 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:42:22.807 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install cephadm 2024-04-25T19:42:22.851 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-04-25T19:42:22.852 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-04-25T19:42:23.340 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:06 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:23.379 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-04-25T19:42:23.462 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-04-25T19:42:23.464 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:42:23.465 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:42:23.465 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-25T19:42:23.465 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:42:23.465 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-25T19:42:23.465 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T19:42:23.465 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:23.465 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-25T19:42:23.465 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:42:23.465 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-04-25T19:42:23.465 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:23.466 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 762 k 2024-04-25T19:42:23.466 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 769 k 2024-04-25T19:42:23.466 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-25T19:42:23.497 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-04-25T19:42:23.544 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-04-25T19:42:23.575 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-04-25T19:42:23.659 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-04-25T19:42:23.778 INFO:teuthology.orchestra.run.smithi029.stdout:cephadm-19.0.0-3291.g994e2f20.el9.noarch.rpm 2.4 MB/s | 762 kB 00:00 2024-04-25T19:42:23.779 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:42:23.779 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.4 MB/s | 762 kB 00:00 2024-04-25T19:42:23.779 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-25T19:42:23.785 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-25T19:42:23.785 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-25T19:42:23.791 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-25T19:42:23.791 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-25T19:42:23.828 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-04-25T19:42:23.861 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-25T19:42:23.928 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-04-25T19:42:23.979 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-04-25T19:42:24.007 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-04-25T19:42:24.040 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9 10/12 2024-04-25T19:42:24.352 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 1/1 2024-04-25T19:42:24.447 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 1/1 2024-04-25T19:42:24.834 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 1/1 2024-04-25T19:42:25.080 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el 11/12 2024-04-25T19:42:25.089 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 1/1 2024-04-25T19:42:25.089 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:25.089 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-25T19:42:25.089 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:42:25.089 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:25.089 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:42:25.115 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noa 12/12 2024-04-25T19:42:25.324 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-immutable-object-cache 2024-04-25T19:42:25.854 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:08 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:25.979 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:42:25.979 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:42:25.979 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-04-25T19:42:25.979 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:42:25.979 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-25T19:42:25.979 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T19:42:25.979 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:25.980 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-25T19:42:25.980 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:42:25.980 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-04-25T19:42:25.980 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:25.980 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 147 k 2024-04-25T19:42:25.980 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 477 k 2024-04-25T19:42:25.980 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-25T19:42:26.302 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-immutable-object-cache-19.0.0-3291.g994e2f 458 kB/s | 147 kB 00:00 2024-04-25T19:42:26.302 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:42:26.303 INFO:teuthology.orchestra.run.smithi029.stdout:Total 455 kB/s | 147 kB 00:00 2024-04-25T19:42:26.303 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-25T19:42:26.311 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-25T19:42:26.311 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-25T19:42:26.372 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-25T19:42:26.372 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-25T19:42:26.412 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noa 12/12 2024-04-25T19:42:26.412 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el 1/12 2024-04-25T19:42:26.412 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noa 2/12 2024-04-25T19:42:26.412 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9 3/12 2024-04-25T19:42:26.412 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-04-25T19:42:26.413 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-04-25T19:42:26.413 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-04-25T19:42:26.413 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-04-25T19:42:26.413 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-04-25T19:42:26.413 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-04-25T19:42:26.413 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-04-25T19:42:26.413 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-04-25T19:42:26.527 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-25T19:42:26.629 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20. 1/1 2024-04-25T19:42:26.654 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20. 1/1 2024-04-25T19:42:26.654 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:42:26.654 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T19:42:26.655 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:26.756 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-04-25T19:42:26.756 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:26.756 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-04-25T19:42:26.756 INFO:teuthology.orchestra.run.smithi102.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T19:42:26.756 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:42:26.756 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:42:26.756 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:42:26.757 INFO:teuthology.orchestra.run.smithi102.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T19:42:26.757 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T19:42:26.757 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T19:42:26.757 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T19:42:26.757 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T19:42:26.757 INFO:teuthology.orchestra.run.smithi102.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T19:42:26.757 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T19:42:26.757 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T19:42:26.757 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:26.757 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:42:27.002 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-25T19:42:27.505 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:10 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:27.580 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20. 1/1 2024-04-25T19:42:27.580 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:27.580 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-25T19:42:27.580 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:42:27.580 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:27.580 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:42:27.627 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:42:27.628 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================= 2024-04-25T19:42:27.628 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-04-25T19:42:27.628 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================= 2024-04-25T19:42:27.628 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-04-25T19:42:27.628 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T19:42:27.628 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-04-25T19:42:27.628 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T19:42:27.628 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T19:42:27.628 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T19:42:27.628 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T19:42:27.628 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T19:42:27.629 INFO:teuthology.orchestra.run.smithi102.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T19:42:27.629 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T19:42:27.629 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T19:42:27.629 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T19:42:27.629 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T19:42:27.629 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T19:42:27.629 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:27.629 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-04-25T19:42:27.629 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================= 2024-04-25T19:42:27.629 INFO:teuthology.orchestra.run.smithi102.stdout:Install 12 Packages 2024-04-25T19:42:27.629 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:27.630 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 38 M 2024-04-25T19:42:27.630 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 209 M 2024-04-25T19:42:27.630 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-04-25T19:42:27.845 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr 2024-04-25T19:42:27.970 INFO:teuthology.orchestra.run.smithi102.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-04-25T19:42:27.995 INFO:teuthology.orchestra.run.smithi102.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.2 MB/s | 803 kB 00:00 2024-04-25T19:42:28.020 INFO:teuthology.orchestra.run.smithi102.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 664 kB/s | 33 kB 00:00 2024-04-25T19:42:28.079 INFO:teuthology.orchestra.run.smithi102.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 318 kB/s | 18 kB 00:00 2024-04-25T19:42:28.138 INFO:teuthology.orchestra.run.smithi102.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 21 MB/s | 3.0 MB 00:00 2024-04-25T19:42:28.163 INFO:teuthology.orchestra.run.smithi102.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 431 kB/s | 35 kB 00:00 2024-04-25T19:42:28.213 INFO:teuthology.orchestra.run.smithi102.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 4.9 MB/s | 244 kB 00:00 2024-04-25T19:42:28.314 INFO:teuthology.orchestra.run.smithi102.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-04-25T19:42:28.375 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:11 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:28.406 INFO:teuthology.orchestra.run.smithi102.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 5.1 MB 00:00 2024-04-25T19:42:28.439 INFO:teuthology.orchestra.run.smithi102.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.0 MB/s | 250 kB 00:00 2024-04-25T19:42:28.460 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 is already installed. 2024-04-25T19:42:28.503 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:42:28.504 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-25T19:42:28.504 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:42:28.584 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-dashboard 2024-04-25T19:42:28.823 INFO:teuthology.orchestra.run.smithi102.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-04-25T19:42:29.112 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:12 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:29.237 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:42:29.238 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:42:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-25T19:42:29.239 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:42:29.239 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-25T19:42:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T19:42:29.239 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-25T19:42:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T19:42:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T19:42:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T19:42:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T19:42:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T19:42:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T19:42:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T19:42:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T19:42:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T19:42:29.240 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T19:42:29.240 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T19:42:29.240 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:29.240 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-25T19:42:29.240 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:42:29.240 INFO:teuthology.orchestra.run.smithi029.stdout:Install 12 Packages 2024-04-25T19:42:29.240 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:29.241 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 8.6 M 2024-04-25T19:42:29.241 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 104 M 2024-04-25T19:42:29.241 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-25T19:42:29.273 INFO:teuthology.orchestra.run.smithi102.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-3 5.0 MB/s | 7.4 MB 00:01 2024-04-25T19:42:29.276 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:42:29.276 INFO:teuthology.orchestra.run.smithi102.stdout:Total 23 MB/s | 38 MB 00:01 2024-04-25T19:42:29.623 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-04-25T19:42:29.650 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-04-25T19:42:29.650 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-04-25T19:42:29.997 INFO:teuthology.orchestra.run.smithi029.stdout:(1/12): ceph-prometheus-alerts-19.0.0-3291.g994 80 kB/s | 16 kB 00:00 2024-04-25T19:42:30.056 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-04-25T19:42:30.056 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-04-25T19:42:30.097 INFO:teuthology.orchestra.run.smithi029.stdout:(2/12): ceph-grafana-dashboards-19.0.0-3291.g99 112 kB/s | 34 kB 00:00 2024-04-25T19:42:30.164 INFO:teuthology.orchestra.run.smithi029.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-04-25T19:42:30.198 INFO:teuthology.orchestra.run.smithi029.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.6 MB/s | 269 kB 00:00 2024-04-25T19:42:30.240 INFO:teuthology.orchestra.run.smithi029.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 11 MB/s | 863 kB 00:00 2024-04-25T19:42:30.416 INFO:teuthology.orchestra.run.smithi029.stdout:(6/12): ceph-mgr-dashboard-19.0.0-3291.g994e2f2 5.7 MB/s | 3.5 MB 00:00 2024-04-25T19:42:30.463 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] abseil-cpp-20211102.0-3.el9.x86_64.rpm: Status code: 404 for https://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/a/abseil-cpp-20211102.0-3.el9.x86_64.rpm (IP: 173.208.208.139) 2024-04-25T19:42:30.464 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] grpc-data-1.46.7-10.el9.noarch.rpm: Status code: 404 for https://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/g/grpc-data-1.46.7-10.el9.noarch.rpm (IP: 173.208.208.139) 2024-04-25T19:42:30.464 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] python3-grpcio-1.46.7-10.el9.x86_64.rpm: Status code: 404 for https://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-grpcio-1.46.7-10.el9.x86_64.rpm (IP: 173.208.208.139) 2024-04-25T19:42:30.551 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] abseil-cpp-20211102.0-3.el9.x86_64.rpm: Status code: 404 for http://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/a/abseil-cpp-20211102.0-3.el9.x86_64.rpm (IP: 173.208.208.139) 2024-04-25T19:42:30.551 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] python3-grpcio-1.46.7-10.el9.x86_64.rpm: Status code: 404 for http://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-grpcio-1.46.7-10.el9.x86_64.rpm (IP: 173.208.208.139) 2024-04-25T19:42:30.558 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] grpc-data-1.46.7-10.el9.noarch.rpm: Status code: 404 for http://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/g/grpc-data-1.46.7-10.el9.noarch.rpm (IP: 173.208.208.139) 2024-04-25T19:42:30.725 INFO:teuthology.orchestra.run.smithi029.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 40 kB/s | 19 kB 00:00 2024-04-25T19:42:30.792 INFO:teuthology.orchestra.run.smithi029.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 926 kB/s | 548 kB 00:00 2024-04-25T19:42:30.885 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-04-25T19:42:30.892 INFO:teuthology.orchestra.run.smithi029.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 4.3 MB/s | 2.0 MB 00:00 2024-04-25T19:42:30.936 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-25T19:42:30.980 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-25T19:42:31.015 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] python3-grpcio-tools-1.46.7-10.el9.x86_64.rpm: Status code: 404 for http://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-grpcio-tools-1.46.7-10.el9.x86_64.rpm (IP: 173.208.208.139) 2024-04-25T19:42:31.061 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] python3-grpcio-tools-1.46.7-10.el9.x86_64.rpm: Status code: 404 for https://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-grpcio-tools-1.46.7-10.el9.x86_64.rpm (IP: 173.208.208.139) 2024-04-25T19:42:31.078 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-25T19:42:31.110 INFO:teuthology.orchestra.run.smithi029.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 376 kB/s | 144 kB 00:00 2024-04-25T19:42:31.110 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] python3-repoze-lru-0.7-16.el9.noarch.rpm: Status code: 404 for http://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-repoze-lru-0.7-16.el9.noarch.rpm (IP: 173.208.208.139) 2024-04-25T19:42:31.122 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-25T19:42:31.152 INFO:teuthology.orchestra.run.smithi029.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 85 kB/s | 31 kB 00:00 2024-04-25T19:42:31.184 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] python3-routes-2.5.1-5.el9.noarch.rpm: Status code: 404 for http://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-routes-2.5.1-5.el9.noarch.rpm (IP: 173.208.208.139) 2024-04-25T19:42:31.227 INFO:teuthology.orchestra.run.smithi029.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 564 kB/s | 188 kB 00:00 2024-04-25T19:42:31.231 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:42:31.231 INFO:teuthology.orchestra.run.smithi029.stdout:Total 4.3 MB/s | 8.6 MB 00:01 2024-04-25T19:42:31.280 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-25T19:42:31.298 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-25T19:42:31.299 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-25T19:42:31.300 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-25T19:42:31.322 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-25T19:42:31.389 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-25T19:42:31.466 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-25T19:42:31.467 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-25T19:42:32.005 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-25T19:42:32.022 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-25T19:42:32.089 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-04-25T19:42:32.133 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-04-25T19:42:32.151 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-25T19:42:32.189 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-04-25T19:42:32.228 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-04-25T19:42:32.318 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-04-25T19:42:32.465 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-04-25T19:42:32.567 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-04-25T19:42:32.623 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-04-25T19:42:32.661 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-04-25T19:42:32.694 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9 10/12 2024-04-25T19:42:33.364 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-25T19:42:33.817 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el 11/12 2024-04-25T19:42:33.854 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noa 12/12 2024-04-25T19:42:34.396 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-25T19:42:34.422 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2 12/12 2024-04-25T19:42:36.184 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noa 12/12 2024-04-25T19:42:36.185 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el 1/12 2024-04-25T19:42:36.185 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noa 2/12 2024-04-25T19:42:36.185 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9 3/12 2024-04-25T19:42:36.185 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-04-25T19:42:36.185 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-04-25T19:42:36.185 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-04-25T19:42:36.185 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-04-25T19:42:36.185 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-04-25T19:42:36.185 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-04-25T19:42:36.185 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-04-25T19:42:36.185 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-04-25T19:42:36.497 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-04-25T19:42:36.497 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:36.497 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-25T19:42:36.497 INFO:teuthology.orchestra.run.smithi029.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T19:42:36.497 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:42:36.497 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:42:36.497 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:42:36.497 INFO:teuthology.orchestra.run.smithi029.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T19:42:36.497 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T19:42:36.497 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T19:42:36.498 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T19:42:36.498 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T19:42:36.498 INFO:teuthology.orchestra.run.smithi029.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T19:42:36.498 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T19:42:36.498 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T19:42:36.498 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:36.498 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:42:36.813 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-25T19:42:36.957 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2 12/12 2024-04-25T19:42:36.957 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2 1/12 2024-04-25T19:42:36.957 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-25T19:42:36.957 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-25T19:42:36.957 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-25T19:42:36.957 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-25T19:42:36.957 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-25T19:42:36.957 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-25T19:42:36.957 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-25T19:42:36.957 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-25T19:42:36.957 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-25T19:42:36.957 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-25T19:42:37.323 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-25T19:42:37.323 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:37.323 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-04-25T19:42:37.323 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:42:37.323 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T19:42:37.323 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T19:42:37.323 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T19:42:37.323 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T19:42:37.323 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T19:42:37.325 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T19:42:37.325 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T19:42:37.325 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T19:42:37.325 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T19:42:37.325 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T19:42:37.325 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T19:42:37.325 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:37.325 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:42:37.344 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:20 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:37.470 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:42:37.473 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-04-25T19:42:37.474 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-25T19:42:37.474 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-04-25T19:42:37.474 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-25T19:42:37.474 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T19:42:37.474 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-25T19:42:37.474 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T19:42:37.474 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T19:42:37.474 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T19:42:37.474 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T19:42:37.474 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T19:42:37.474 INFO:teuthology.orchestra.run.smithi029.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T19:42:37.474 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T19:42:37.474 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T19:42:37.474 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T19:42:37.475 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T19:42:37.475 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T19:42:37.475 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:37.475 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-25T19:42:37.475 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-04-25T19:42:37.475 INFO:teuthology.orchestra.run.smithi029.stdout:Install 12 Packages 2024-04-25T19:42:37.475 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:37.475 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 38 M 2024-04-25T19:42:37.475 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 209 M 2024-04-25T19:42:37.475 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-25T19:42:37.708 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-rook 2024-04-25T19:42:37.841 INFO:teuthology.orchestra.run.smithi029.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-04-25T19:42:37.883 INFO:teuthology.orchestra.run.smithi029.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.7 MB/s | 803 kB 00:00 2024-04-25T19:42:37.910 INFO:teuthology.orchestra.run.smithi029.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 480 kB/s | 33 kB 00:00 2024-04-25T19:42:38.002 INFO:teuthology.orchestra.run.smithi029.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 25 MB/s | 3.0 MB 00:00 2024-04-25T19:42:38.027 INFO:teuthology.orchestra.run.smithi029.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 159 kB/s | 18 kB 00:00 2024-04-25T19:42:38.053 INFO:teuthology.orchestra.run.smithi029.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 695 kB/s | 35 kB 00:00 2024-04-25T19:42:38.187 INFO:teuthology.orchestra.run.smithi029.stdout:(7/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-04-25T19:42:38.221 INFO:teuthology.orchestra.run.smithi029.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.4 MB/s | 244 kB 00:00 2024-04-25T19:42:38.224 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:21 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:38.344 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:42:38.346 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:38.346 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-04-25T19:42:38.346 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:38.346 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-04-25T19:42:38.346 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T19:42:38.346 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-04-25T19:42:38.347 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T19:42:38.347 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T19:42:38.347 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T19:42:38.347 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T19:42:38.347 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T19:42:38.347 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T19:42:38.348 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T19:42:38.348 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T19:42:38.348 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T19:42:38.348 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T19:42:38.348 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T19:42:38.349 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T19:42:38.349 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:38.349 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-04-25T19:42:38.349 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:38.349 INFO:teuthology.orchestra.run.smithi102.stdout:Install 13 Packages 2024-04-25T19:42:38.349 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:38.350 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 2.2 M 2024-04-25T19:42:38.351 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 26 M 2024-04-25T19:42:38.351 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-04-25T19:42:38.371 INFO:teuthology.orchestra.run.smithi029.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 28 MB/s | 5.1 MB 00:00 2024-04-25T19:42:38.397 INFO:teuthology.orchestra.run.smithi029.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.4 MB/s | 250 kB 00:00 2024-04-25T19:42:38.647 INFO:teuthology.orchestra.run.smithi029.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-3 7.6 MB/s | 7.4 MB 00:00 2024-04-25T19:42:38.964 INFO:teuthology.orchestra.run.smithi029.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-04-25T19:42:38.969 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:42:38.969 INFO:teuthology.orchestra.run.smithi029.stdout:Total 25 MB/s | 38 MB 00:01 2024-04-25T19:42:39.007 INFO:teuthology.orchestra.run.smithi102.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 203 kB/s | 26 kB 00:00 2024-04-25T19:42:39.023 INFO:teuthology.orchestra.run.smithi102.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 134 kB/s | 19 kB 00:00 2024-04-25T19:42:39.074 INFO:teuthology.orchestra.run.smithi102.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 3.3 MB/s | 222 kB 00:00 2024-04-25T19:42:39.099 INFO:teuthology.orchestra.run.smithi102.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-04-25T19:42:39.124 INFO:teuthology.orchestra.run.smithi102.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 5.4 MB/s | 279 kB 00:00 2024-04-25T19:42:39.141 INFO:teuthology.orchestra.run.smithi102.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.3 MB/s | 54 kB 00:00 2024-04-25T19:42:39.167 INFO:teuthology.orchestra.run.smithi102.stdout:(7/13): ceph-mgr-rook-19.0.0-3291.g994e2f20.el9 172 kB/s | 49 kB 00:00 2024-04-25T19:42:39.327 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-25T19:42:39.334 INFO:teuthology.orchestra.run.smithi102.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 78 kB/s | 15 kB 00:00 2024-04-25T19:42:39.350 INFO:teuthology.orchestra.run.smithi102.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 142 kB/s | 32 kB 00:00 2024-04-25T19:42:39.356 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-25T19:42:39.356 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-25T19:42:39.543 INFO:teuthology.orchestra.run.smithi102.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 308 kB/s | 59 kB 00:00 2024-04-25T19:42:39.576 INFO:teuthology.orchestra.run.smithi102.stdout:(11/13): python3-google-auth-2.29.0-1.el9.noarc 503 kB/s | 206 kB 00:00 2024-04-25T19:42:39.676 INFO:teuthology.orchestra.run.smithi102.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 672 kB/s | 90 kB 00:00 2024-04-25T19:42:39.751 INFO:teuthology.orchestra.run.smithi102.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.4 MB/s | 1.0 MB 00:00 2024-04-25T19:42:39.753 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:42:39.754 INFO:teuthology.orchestra.run.smithi102.stdout:Total 1.6 MB/s | 2.2 MB 00:01 2024-04-25T19:42:39.793 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-25T19:42:39.794 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-25T19:42:39.815 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-04-25T19:42:39.830 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-04-25T19:42:39.830 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-04-25T19:42:39.914 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-04-25T19:42:39.915 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-04-25T19:42:40.418 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-04-25T19:42:40.589 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-25T19:42:40.661 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-25T19:42:40.697 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-25T19:42:40.724 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-25T19:42:40.762 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-25T19:42:40.763 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-25T19:42:40.804 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-25T19:42:40.806 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-25T19:42:40.907 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-25T19:42:40.913 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-25T19:42:40.960 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-25T19:42:41.016 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-04-25T19:42:41.063 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-25T19:42:41.146 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-25T19:42:41.177 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-25T19:42:41.256 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-25T19:42:41.779 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-25T19:42:41.836 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-25T19:42:41.870 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-25T19:42:41.918 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-25T19:42:41.940 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 13/13 2024-04-25T19:42:41.950 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-25T19:42:42.085 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-25T19:42:42.676 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 13/13 2024-04-25T19:42:42.676 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 1/13 2024-04-25T19:42:42.676 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-25T19:42:42.676 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-25T19:42:42.676 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-25T19:42:42.676 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-25T19:42:42.677 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-25T19:42:42.677 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-25T19:42:42.677 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-25T19:42:42.677 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-25T19:42:42.677 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-04-25T19:42:42.677 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-25T19:42:42.677 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-25T19:42:42.979 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-25T19:42:42.979 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:42.979 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-04-25T19:42:42.979 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:42:42.979 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T19:42:42.979 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T19:42:42.979 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T19:42:42.979 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T19:42:42.979 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T19:42:42.979 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T19:42:42.980 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T19:42:42.980 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T19:42:42.980 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T19:42:42.980 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T19:42:42.980 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T19:42:42.980 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T19:42:42.980 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:42.980 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:42:43.216 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-cephadm 2024-04-25T19:42:43.395 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-25T19:42:43.723 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:26 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:43.842 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:42:43.843 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:43.843 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-04-25T19:42:43.843 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:43.843 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-04-25T19:42:43.843 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T19:42:43.843 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-04-25T19:42:43.843 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T19:42:43.843 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T19:42:43.843 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T19:42:43.844 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T19:42:43.844 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T19:42:43.844 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:43.844 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-04-25T19:42:43.844 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:43.844 INFO:teuthology.orchestra.run.smithi102.stdout:Install 6 Packages 2024-04-25T19:42:43.844 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:43.845 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 7.0 M 2024-04-25T19:42:43.845 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 33 M 2024-04-25T19:42:43.845 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-04-25T19:42:44.346 INFO:teuthology.orchestra.run.smithi102.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-04-25T19:42:44.413 INFO:teuthology.orchestra.run.smithi102.stdout:(2/6): ceph-mgr-cephadm-19.0.0-3291.g994e2f20.e 553 kB/s | 142 kB 00:00 2024-04-25T19:42:44.506 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-25T19:42:44.527 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2 12/12 2024-04-25T19:42:44.547 INFO:teuthology.orchestra.run.smithi102.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-04-25T19:42:44.905 INFO:teuthology.orchestra.run.smithi102.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 118 kB/s | 58 kB 00:00 2024-04-25T19:42:44.999 INFO:teuthology.orchestra.run.smithi102.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 119 kB/s | 51 kB 00:00 2024-04-25T19:42:45.041 INFO:teuthology.orchestra.run.smithi102.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 790 kB/s | 548 kB 00:00 2024-04-25T19:42:45.045 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:42:45.045 INFO:teuthology.orchestra.run.smithi102.stdout:Total 5.8 MB/s | 7.0 MB 00:01 2024-04-25T19:42:45.115 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-04-25T19:42:45.127 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-04-25T19:42:45.127 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-04-25T19:42:45.223 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-04-25T19:42:45.223 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-04-25T19:42:45.526 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-04-25T19:42:45.690 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-25T19:42:45.743 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-25T19:42:46.263 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-25T19:42:46.354 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-25T19:42:46.417 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-25T19:42:46.455 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 6/6 2024-04-25T19:42:47.236 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 6/6 2024-04-25T19:42:47.236 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 1/6 2024-04-25T19:42:47.236 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-25T19:42:47.236 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-25T19:42:47.236 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-25T19:42:47.236 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-25T19:42:47.490 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-25T19:42:47.490 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:47.490 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-04-25T19:42:47.490 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:42:47.490 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T19:42:47.491 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T19:42:47.491 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T19:42:47.491 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T19:42:47.491 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T19:42:47.491 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:47.491 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:42:47.738 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-fuse 2024-04-25T19:42:48.255 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:31 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:48.374 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:42:48.375 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:48.375 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-04-25T19:42:48.375 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:48.375 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-04-25T19:42:48.375 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T19:42:48.375 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-04-25T19:42:48.375 INFO:teuthology.orchestra.run.smithi102.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T19:42:48.375 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:48.376 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-04-25T19:42:48.376 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:48.376 INFO:teuthology.orchestra.run.smithi102.stdout:Install 2 Packages 2024-04-25T19:42:48.376 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:48.376 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 943 k 2024-04-25T19:42:48.376 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 3.1 M 2024-04-25T19:42:48.376 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-04-25T19:42:48.518 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2 12/12 2024-04-25T19:42:48.518 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2 1/12 2024-04-25T19:42:48.518 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-25T19:42:48.518 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-25T19:42:48.518 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-25T19:42:48.519 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-25T19:42:48.519 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-25T19:42:48.519 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-25T19:42:48.519 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-25T19:42:48.519 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-25T19:42:48.519 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-25T19:42:48.519 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-25T19:42:48.785 INFO:teuthology.orchestra.run.smithi102.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 615 kB/s | 80 kB 00:00 2024-04-25T19:42:48.948 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-25T19:42:48.948 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:48.948 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-25T19:42:48.948 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:42:48.948 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T19:42:48.949 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T19:42:48.949 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T19:42:48.949 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T19:42:48.949 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T19:42:48.949 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T19:42:48.949 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T19:42:48.949 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T19:42:48.949 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T19:42:48.949 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T19:42:48.949 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T19:42:48.949 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:48.949 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:42:48.968 INFO:teuthology.orchestra.run.smithi102.stdout:(2/2): ceph-fuse-19.0.0-3291.g994e2f20.el9.x86_ 2.7 MB/s | 863 kB 00:00 2024-04-25T19:42:48.969 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:42:48.969 INFO:teuthology.orchestra.run.smithi102.stdout:Total 1.6 MB/s | 943 kB 00:00 2024-04-25T19:42:48.981 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-04-25T19:42:49.003 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-04-25T19:42:49.003 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-04-25T19:42:49.080 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-04-25T19:42:49.081 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-04-25T19:42:49.259 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-04-25T19:42:49.402 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-rook 2024-04-25T19:42:49.404 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-25T19:42:49.475 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2/2 2024-04-25T19:42:49.937 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:32 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:50.062 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:42:50.063 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:42:50.064 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-25T19:42:50.064 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:42:50.064 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-25T19:42:50.064 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T19:42:50.064 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-25T19:42:50.064 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T19:42:50.064 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T19:42:50.064 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T19:42:50.064 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T19:42:50.064 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T19:42:50.064 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T19:42:50.065 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T19:42:50.065 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T19:42:50.065 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T19:42:50.065 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T19:42:50.065 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T19:42:50.065 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T19:42:50.065 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:50.065 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-25T19:42:50.065 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:42:50.065 INFO:teuthology.orchestra.run.smithi029.stdout:Install 13 Packages 2024-04-25T19:42:50.065 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:50.066 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 2.2 M 2024-04-25T19:42:50.066 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 26 M 2024-04-25T19:42:50.066 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-25T19:42:50.145 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2/2 2024-04-25T19:42:50.145 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 1/2 2024-04-25T19:42:50.398 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-25T19:42:50.398 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:50.398 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-04-25T19:42:50.398 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-25T19:42:50.398 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:50.398 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:42:50.490 INFO:teuthology.orchestra.run.smithi029.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 186 kB/s | 26 kB 00:00 2024-04-25T19:42:50.516 INFO:teuthology.orchestra.run.smithi029.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 117 kB/s | 19 kB 00:00 2024-04-25T19:42:50.600 INFO:teuthology.orchestra.run.smithi029.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-04-25T19:42:50.626 INFO:teuthology.orchestra.run.smithi029.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-04-25T19:42:50.662 INFO:teuthology.orchestra.run.smithi029.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.6 MB/s | 279 kB 00:00 2024-04-25T19:42:50.675 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-volume 2024-04-25T19:42:50.685 INFO:teuthology.orchestra.run.smithi029.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 917 kB/s | 54 kB 00:00 2024-04-25T19:42:50.710 INFO:teuthology.orchestra.run.smithi029.stdout:(7/13): ceph-mgr-rook-19.0.0-3291.g994e2f20.el9 138 kB/s | 49 kB 00:00 2024-04-25T19:42:50.794 INFO:teuthology.orchestra.run.smithi029.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 137 kB/s | 15 kB 00:00 2024-04-25T19:42:50.820 INFO:teuthology.orchestra.run.smithi029.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 201 kB/s | 32 kB 00:00 2024-04-25T19:42:50.870 INFO:teuthology.orchestra.run.smithi029.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 1.3 MB/s | 206 kB 00:00 2024-04-25T19:42:50.896 INFO:teuthology.orchestra.run.smithi029.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 780 kB/s | 59 kB 00:00 2024-04-25T19:42:50.929 INFO:teuthology.orchestra.run.smithi029.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-04-25T19:42:51.013 INFO:teuthology.orchestra.run.smithi029.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.7 MB/s | 1.0 MB 00:00 2024-04-25T19:42:51.015 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:42:51.015 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.3 MB/s | 2.2 MB 00:00 2024-04-25T19:42:51.080 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-25T19:42:51.096 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-25T19:42:51.096 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-25T19:42:51.188 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-25T19:42:51.188 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-25T19:42:51.200 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:34 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:51.321 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:42:51.322 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:51.322 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-04-25T19:42:51.322 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:51.322 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-04-25T19:42:51.322 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T19:42:51.322 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-04-25T19:42:51.322 INFO:teuthology.orchestra.run.smithi102.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T19:42:51.322 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:51.322 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-04-25T19:42:51.322 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:51.322 INFO:teuthology.orchestra.run.smithi102.stdout:Install 2 Packages 2024-04-25T19:42:51.323 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:51.323 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 350 k 2024-04-25T19:42:51.323 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 1.5 M 2024-04-25T19:42:51.323 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-04-25T19:42:51.564 INFO:teuthology.orchestra.run.smithi102.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 645 kB/s | 77 kB 00:00 2024-04-25T19:42:51.715 INFO:teuthology.orchestra.run.smithi102.stdout:(2/2): ceph-volume-19.0.0-3291.g994e2f20.el9.no 1.0 MB/s | 273 kB 00:00 2024-04-25T19:42:51.715 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:42:51.716 INFO:teuthology.orchestra.run.smithi102.stdout:Total 890 kB/s | 350 kB 00:00 2024-04-25T19:42:51.722 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-25T19:42:51.727 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-04-25T19:42:51.737 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-04-25T19:42:51.738 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-04-25T19:42:51.763 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-04-25T19:42:51.763 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-04-25T19:42:51.897 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-25T19:42:51.903 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-04-25T19:42:51.962 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-25T19:42:52.035 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-25T19:42:52.071 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-25T19:42:52.077 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-25T19:42:52.109 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2/2 2024-04-25T19:42:52.122 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-25T19:42:52.124 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2/2 2024-04-25T19:42:52.124 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:42:52.124 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T19:42:52.124 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:52.237 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-25T19:42:52.360 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-04-25T19:42:52.420 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-25T19:42:52.815 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 1/2 2024-04-25T19:42:53.081 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-25T19:42:53.081 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:53.081 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-04-25T19:42:53.081 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:42:53.081 INFO:teuthology.orchestra.run.smithi102.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T19:42:53.081 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:53.081 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:42:53.192 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-25T19:42:53.255 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-25T19:42:53.289 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-25T19:42:53.347 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-25T19:42:53.376 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 13/13 2024-04-25T19:42:53.420 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librados-devel 2024-04-25T19:42:53.942 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:36 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:54.065 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:42:54.065 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:54.066 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-04-25T19:42:54.066 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:54.066 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-04-25T19:42:54.066 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T19:42:54.066 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:54.066 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-04-25T19:42:54.066 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:54.066 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-04-25T19:42:54.066 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:54.066 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 127 k 2024-04-25T19:42:54.066 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 457 k 2024-04-25T19:42:54.066 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-04-25T19:42:54.271 INFO:teuthology.orchestra.run.smithi102.stdout:librados-devel-19.0.0-3291.g994e2f20.el9.x86_64 622 kB/s | 127 kB 00:00 2024-04-25T19:42:54.271 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:42:54.271 INFO:teuthology.orchestra.run.smithi102.stdout:Total 617 kB/s | 127 kB 00:00 2024-04-25T19:42:54.272 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-04-25T19:42:54.277 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-04-25T19:42:54.277 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-04-25T19:42:54.340 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-04-25T19:42:54.341 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-04-25T19:42:54.444 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 13/13 2024-04-25T19:42:54.444 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 1/13 2024-04-25T19:42:54.444 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-25T19:42:54.444 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-25T19:42:54.444 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-25T19:42:54.444 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-25T19:42:54.445 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-25T19:42:54.445 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-25T19:42:54.445 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-25T19:42:54.445 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-25T19:42:54.445 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-04-25T19:42:54.445 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-25T19:42:54.445 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-25T19:42:54.504 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-04-25T19:42:54.601 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:42:54.755 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-25T19:42:54.755 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:54.755 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-25T19:42:54.755 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:42:54.755 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T19:42:54.755 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T19:42:54.755 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T19:42:54.755 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T19:42:54.755 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T19:42:54.755 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T19:42:54.756 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T19:42:54.756 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T19:42:54.756 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T19:42:54.756 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T19:42:54.756 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T19:42:54.756 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T19:42:54.756 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:54.756 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:42:55.021 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:42:55.056 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-cephadm 2024-04-25T19:42:55.267 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:42:55.267 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:55.267 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-04-25T19:42:55.267 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:42:55.267 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:55.267 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:42:55.478 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install libcephfs2 2024-04-25T19:42:55.589 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:38 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:55.713 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:42:55.714 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:42:55.714 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-25T19:42:55.714 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:42:55.714 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-25T19:42:55.714 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T19:42:55.714 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-25T19:42:55.714 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T19:42:55.714 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T19:42:55.714 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T19:42:55.714 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T19:42:55.714 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T19:42:55.714 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:55.715 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-25T19:42:55.715 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:42:55.715 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2024-04-25T19:42:55.715 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:55.715 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 7.0 M 2024-04-25T19:42:55.715 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 33 M 2024-04-25T19:42:55.715 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-25T19:42:56.006 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:39 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:56.086 INFO:teuthology.orchestra.run.smithi102.stdout:Package libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 is already installed. 2024-04-25T19:42:56.128 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:42:56.129 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-04-25T19:42:56.129 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:42:56.219 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install libcephfs-devel 2024-04-25T19:42:56.232 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-04-25T19:42:56.424 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-04-25T19:42:56.525 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): ceph-mgr-cephadm-19.0.0-3291.g994e2f20.e 303 kB/s | 142 kB 00:00 2024-04-25T19:42:56.567 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 408 kB/s | 58 kB 00:00 2024-04-25T19:42:56.634 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2024-04-25T19:42:56.718 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 265 kB/s | 51 kB 00:00 2024-04-25T19:42:56.721 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:42:56.721 INFO:teuthology.orchestra.run.smithi029.stdout:Total 6.9 MB/s | 7.0 MB 00:01 2024-04-25T19:42:56.750 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:39 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:56.794 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-25T19:42:56.806 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-25T19:42:56.807 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-25T19:42:56.869 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:42:56.870 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:56.870 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-04-25T19:42:56.870 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:56.870 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-04-25T19:42:56.870 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T19:42:56.870 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:56.870 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-04-25T19:42:56.870 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:42:56.871 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-04-25T19:42:56.871 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:56.871 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 32 k 2024-04-25T19:42:56.871 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 142 k 2024-04-25T19:42:56.871 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-04-25T19:42:56.910 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-25T19:42:56.910 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-25T19:42:57.032 INFO:teuthology.orchestra.run.smithi102.stdout:libcephfs-devel-19.0.0-3291.g994e2f20.el9.x86_6 197 kB/s | 32 kB 00:00 2024-04-25T19:42:57.032 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:42:57.033 INFO:teuthology.orchestra.run.smithi102.stdout:Total 195 kB/s | 32 kB 00:00 2024-04-25T19:42:57.033 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-04-25T19:42:57.036 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-04-25T19:42:57.036 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-04-25T19:42:57.038 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-04-25T19:42:57.039 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-04-25T19:42:57.124 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-04-25T19:42:57.224 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-25T19:42:57.228 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:42:57.418 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-25T19:42:57.473 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-25T19:42:57.594 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:42:57.810 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:42:57.810 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:57.810 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-04-25T19:42:57.810 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:42:57.811 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:42:57.811 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:42:57.972 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librados2 2024-04-25T19:42:58.040 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-25T19:42:58.140 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-25T19:42:58.211 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-25T19:42:58.247 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 6/6 2024-04-25T19:42:58.489 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:41 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:58.568 INFO:teuthology.orchestra.run.smithi102.stdout:Package librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 is already installed. 2024-04-25T19:42:58.609 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:42:58.610 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-04-25T19:42:58.610 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:42:58.673 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librbd1 2024-04-25T19:42:59.192 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:42 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:59.272 INFO:teuthology.orchestra.run.smithi102.stdout:Package librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 is already installed. 2024-04-25T19:42:59.314 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:42:59.315 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-04-25T19:42:59.315 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:42:59.326 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 6/6 2024-04-25T19:42:59.326 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 1/6 2024-04-25T19:42:59.326 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-25T19:42:59.326 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-25T19:42:59.326 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-25T19:42:59.326 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-25T19:42:59.377 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rados 2024-04-25T19:42:59.587 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-25T19:42:59.587 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:59.587 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-25T19:42:59.587 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:42:59.587 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T19:42:59.587 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T19:42:59.587 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T19:42:59.587 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T19:42:59.587 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T19:42:59.587 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:42:59.587 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:42:59.888 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-fuse 2024-04-25T19:42:59.889 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:42 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:42:59.968 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 is already installed. 2024-04-25T19:43:00.010 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:43:00.011 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-04-25T19:43:00.011 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:43:00.076 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rgw 2024-04-25T19:43:00.429 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:43 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:43:00.554 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:43:00.555 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:43:00.555 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-25T19:43:00.555 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:43:00.555 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-25T19:43:00.555 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T19:43:00.555 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-25T19:43:00.555 INFO:teuthology.orchestra.run.smithi029.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T19:43:00.555 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:00.555 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-25T19:43:00.555 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:43:00.555 INFO:teuthology.orchestra.run.smithi029.stdout:Install 2 Packages 2024-04-25T19:43:00.555 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:00.556 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 943 k 2024-04-25T19:43:00.556 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 3.1 M 2024-04-25T19:43:00.556 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-25T19:43:00.586 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:43 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:43:00.665 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 is already installed. 2024-04-25T19:43:00.706 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:43:00.707 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-04-25T19:43:00.708 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:43:00.772 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-cephfs 2024-04-25T19:43:00.824 INFO:teuthology.orchestra.run.smithi029.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 574 kB/s | 80 kB 00:00 2024-04-25T19:43:01.008 INFO:teuthology.orchestra.run.smithi029.stdout:(2/2): ceph-fuse-19.0.0-3291.g994e2f20.el9.x86_ 2.6 MB/s | 863 kB 00:00 2024-04-25T19:43:01.009 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:43:01.009 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.0 MB/s | 943 kB 00:00 2024-04-25T19:43:01.021 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-25T19:43:01.043 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-25T19:43:01.044 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-25T19:43:01.126 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-25T19:43:01.126 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-25T19:43:01.285 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:44 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:43:01.316 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-25T19:43:01.364 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 is already installed. 2024-04-25T19:43:01.405 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:43:01.406 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-04-25T19:43:01.406 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:43:01.468 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-25T19:43:01.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rbd 2024-04-25T19:43:01.553 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2/2 2024-04-25T19:43:01.996 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:44 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:43:02.075 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 is already installed. 2024-04-25T19:43:02.117 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:43:02.118 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-04-25T19:43:02.118 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:43:02.180 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-fuse 2024-04-25T19:43:02.252 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2/2 2024-04-25T19:43:02.252 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 1/2 2024-04-25T19:43:02.542 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-25T19:43:02.542 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:02.542 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-25T19:43:02.542 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-25T19:43:02.542 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:02.542 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:43:02.698 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:45 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:43:02.817 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:43:02.818 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:43:02.818 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-04-25T19:43:02.818 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:43:02.818 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-04-25T19:43:02.818 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T19:43:02.818 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:02.818 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-04-25T19:43:02.818 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:43:02.818 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-04-25T19:43:02.819 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:02.819 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 86 k 2024-04-25T19:43:02.819 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 256 k 2024-04-25T19:43:02.819 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-04-25T19:43:02.918 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-volume 2024-04-25T19:43:03.027 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-fuse-19.0.0-3291.g994e2f20.el9.x86_64.rpm 415 kB/s | 86 kB 00:00 2024-04-25T19:43:03.027 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:43:03.027 INFO:teuthology.orchestra.run.smithi102.stdout:Total 412 kB/s | 86 kB 00:00 2024-04-25T19:43:03.028 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-04-25T19:43:03.035 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-04-25T19:43:03.035 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-04-25T19:43:03.101 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-04-25T19:43:03.102 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-04-25T19:43:03.249 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-04-25T19:43:03.364 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:03.460 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:46 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:43:03.588 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:43:03.589 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:43:03.589 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-25T19:43:03.589 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:43:03.589 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-25T19:43:03.589 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T19:43:03.590 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-25T19:43:03.590 INFO:teuthology.orchestra.run.smithi029.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T19:43:03.590 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:03.590 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-25T19:43:03.590 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:43:03.590 INFO:teuthology.orchestra.run.smithi029.stdout:Install 2 Packages 2024-04-25T19:43:03.590 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:03.590 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 350 k 2024-04-25T19:43:03.590 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 1.5 M 2024-04-25T19:43:03.590 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-25T19:43:03.707 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:03.881 INFO:teuthology.orchestra.run.smithi029.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 554 kB/s | 77 kB 00:00 2024-04-25T19:43:03.944 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:03.944 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:03.944 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-04-25T19:43:03.944 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:43:03.944 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:03.944 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:43:04.157 INFO:teuthology.orchestra.run.smithi029.stdout:(2/2): ceph-volume-19.0.0-3291.g994e2f20.el9.no 658 kB/s | 273 kB 00:00 2024-04-25T19:43:04.158 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:43:04.158 INFO:teuthology.orchestra.run.smithi029.stdout:Total 616 kB/s | 350 kB 00:00 2024-04-25T19:43:04.170 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-25T19:43:04.180 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-25T19:43:04.181 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-25T19:43:04.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-mirror 2024-04-25T19:43:04.208 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-25T19:43:04.209 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-25T19:43:04.360 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-25T19:43:04.566 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-25T19:43:04.613 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2/2 2024-04-25T19:43:04.630 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2/2 2024-04-25T19:43:04.630 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:43:04.630 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T19:43:04.630 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:04.714 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:47 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:43:04.838 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:43:04.839 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:43:04.839 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-04-25T19:43:04.839 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:43:04.839 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-04-25T19:43:04.839 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T19:43:04.839 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:04.839 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-04-25T19:43:04.839 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:43:04.839 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-04-25T19:43:04.839 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:04.840 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 3.0 M 2024-04-25T19:43:04.840 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 14 M 2024-04-25T19:43:04.840 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-04-25T19:43:05.385 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 1/2 2024-04-25T19:43:05.684 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-25T19:43:05.685 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:05.685 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-25T19:43:05.685 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:43:05.685 INFO:teuthology.orchestra.run.smithi029.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T19:43:05.685 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:05.685 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:43:06.059 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados-devel 2024-04-25T19:43:06.339 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-mirror-19.0.0-3291.g994e2f20.el9.x86_64.rpm 2.0 MB/s | 3.0 MB 00:01 2024-04-25T19:43:06.339 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:43:06.340 INFO:teuthology.orchestra.run.smithi102.stdout:Total 2.0 MB/s | 3.0 MB 00:01 2024-04-25T19:43:06.340 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-04-25T19:43:06.349 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-04-25T19:43:06.349 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-04-25T19:43:06.429 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-04-25T19:43:06.429 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-04-25T19:43:06.598 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:49 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:43:06.725 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:43:06.726 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:43:06.726 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-04-25T19:43:06.726 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:43:06.726 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-25T19:43:06.726 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T19:43:06.726 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:06.726 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-25T19:43:06.726 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:43:06.726 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-04-25T19:43:06.726 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:06.726 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 127 k 2024-04-25T19:43:06.726 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 457 k 2024-04-25T19:43:06.726 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-25T19:43:06.874 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-04-25T19:43:06.931 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:06.956 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:06.956 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:43:06.956 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T19:43:06.956 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T19:43:06.956 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T19:43:06.956 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:07.065 INFO:teuthology.orchestra.run.smithi029.stdout:librados-devel-19.0.0-3291.g994e2f20.el9.x86_64 376 kB/s | 127 kB 00:00 2024-04-25T19:43:07.065 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:43:07.065 INFO:teuthology.orchestra.run.smithi029.stdout:Total 374 kB/s | 127 kB 00:00 2024-04-25T19:43:07.065 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-25T19:43:07.072 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-25T19:43:07.072 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-25T19:43:07.142 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-25T19:43:07.143 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-25T19:43:07.293 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-25T19:43:07.398 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:07.888 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:08.006 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-04-25T19:43:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:43:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:08.007 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:43:08.143 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:08.143 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:08.143 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-25T19:43:08.143 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:43:08.143 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:08.143 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:43:08.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-nbd 2024-04-25T19:43:08.441 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs2 2024-04-25T19:43:08.742 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:51 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:43:08.868 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T19:43:08.869 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:43:08.869 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-04-25T19:43:08.869 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:43:08.869 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-04-25T19:43:08.870 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T19:43:08.870 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:08.870 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-04-25T19:43:08.870 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-04-25T19:43:08.870 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-04-25T19:43:08.870 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:08.870 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 169 k 2024-04-25T19:43:08.870 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 540 k 2024-04-25T19:43:08.870 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-04-25T19:43:08.982 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:51 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:43:09.063 INFO:teuthology.orchestra.run.smithi029.stdout:Package libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 is already installed. 2024-04-25T19:43:09.106 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:43:09.107 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-25T19:43:09.107 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:43:09.193 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-nbd-19.0.0-3291.g994e2f20.el9.x86_64.rpm 526 kB/s | 169 kB 00:00 2024-04-25T19:43:09.193 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:43:09.193 INFO:teuthology.orchestra.run.smithi102.stdout:Total 524 kB/s | 169 kB 00:00 2024-04-25T19:43:09.193 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-04-25T19:43:09.201 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-04-25T19:43:09.201 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-04-25T19:43:09.206 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs-devel 2024-04-25T19:43:09.265 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-04-25T19:43:09.265 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-04-25T19:43:09.415 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-04-25T19:43:09.529 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:09.748 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:52 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:43:09.874 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:43:09.874 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:43:09.874 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-04-25T19:43:09.875 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:43:09.875 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-25T19:43:09.875 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T19:43:09.875 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:09.875 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-25T19:43:09.875 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:43:09.875 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-04-25T19:43:09.875 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:09.875 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 32 k 2024-04-25T19:43:09.875 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 142 k 2024-04-25T19:43:09.875 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-25T19:43:09.932 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:10.220 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:10.220 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:10.220 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-04-25T19:43:10.220 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:43:10.220 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:10.220 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T19:43:10.392 INFO:teuthology.orchestra.run.smithi029.stdout:libcephfs-devel-19.0.0-3291.g994e2f20.el9.x86_6 61 kB/s | 32 kB 00:00 2024-04-25T19:43:10.393 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:43:10.393 INFO:teuthology.orchestra.run.smithi029.stdout:Total 61 kB/s | 32 kB 00:00 2024-04-25T19:43:10.393 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-25T19:43:10.397 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-25T19:43:10.397 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-25T19:43:10.400 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-25T19:43:10.400 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-25T19:43:10.445 DEBUG:teuthology.parallel:result is None 2024-04-25T19:43:10.486 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-25T19:43:10.618 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:11.028 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:11.249 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:11.249 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:11.249 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-25T19:43:11.249 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:43:11.249 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:11.249 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:43:11.497 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados2 2024-04-25T19:43:12.028 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:55 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:43:12.109 INFO:teuthology.orchestra.run.smithi029.stdout:Package librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 is already installed. 2024-04-25T19:43:12.153 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:43:12.154 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-25T19:43:12.154 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:43:12.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librbd1 2024-04-25T19:43:12.783 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:55 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:43:12.865 INFO:teuthology.orchestra.run.smithi029.stdout:Package librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 is already installed. 2024-04-25T19:43:12.909 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:43:12.910 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-25T19:43:12.910 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:43:12.997 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rados 2024-04-25T19:43:13.533 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:56 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:43:13.614 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 is already installed. 2024-04-25T19:43:13.656 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:43:13.658 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-25T19:43:13.658 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:43:13.748 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rgw 2024-04-25T19:43:14.283 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:57 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:43:14.364 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 is already installed. 2024-04-25T19:43:14.407 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:43:14.408 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-25T19:43:14.408 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:43:14.497 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-cephfs 2024-04-25T19:43:15.032 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:58 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:43:15.114 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 is already installed. 2024-04-25T19:43:15.157 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:43:15.158 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-25T19:43:15.158 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:43:15.251 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rbd 2024-04-25T19:43:15.791 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:58 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:43:15.873 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 is already installed. 2024-04-25T19:43:15.917 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:43:15.918 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-25T19:43:15.918 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:43:16.009 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-fuse 2024-04-25T19:43:16.552 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:59 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:43:16.678 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:43:16.678 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:43:16.679 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-04-25T19:43:16.679 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:43:16.679 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-25T19:43:16.679 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T19:43:16.679 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:16.679 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-25T19:43:16.679 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:43:16.679 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-04-25T19:43:16.679 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:16.680 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 86 k 2024-04-25T19:43:16.680 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 256 k 2024-04-25T19:43:16.680 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-25T19:43:17.050 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-fuse-19.0.0-3291.g994e2f20.el9.x86_64.rpm 232 kB/s | 86 kB 00:00 2024-04-25T19:43:17.051 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:43:17.051 INFO:teuthology.orchestra.run.smithi029.stdout:Total 232 kB/s | 86 kB 00:00 2024-04-25T19:43:17.051 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-25T19:43:17.059 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-25T19:43:17.059 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-25T19:43:17.131 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-25T19:43:17.132 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-25T19:43:17.284 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-25T19:43:17.384 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:17.791 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:18.041 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:18.041 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:18.041 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-25T19:43:18.041 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:43:18.041 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:18.041 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:43:18.360 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-mirror 2024-04-25T19:43:18.904 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:01 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:43:19.029 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:43:19.030 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:43:19.030 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-04-25T19:43:19.030 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:43:19.030 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-25T19:43:19.030 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T19:43:19.030 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:19.030 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-25T19:43:19.031 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:43:19.031 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-04-25T19:43:19.031 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:19.031 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 3.0 M 2024-04-25T19:43:19.031 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 14 M 2024-04-25T19:43:19.031 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-25T19:43:22.067 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-mirror-19.0.0-3291.g994e2f20.el9.x86_64.rpm 1.0 MB/s | 3.0 MB 00:03 2024-04-25T19:43:22.068 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:43:22.069 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.0 MB/s | 3.0 MB 00:03 2024-04-25T19:43:22.069 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-25T19:43:22.081 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-25T19:43:22.081 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-25T19:43:22.169 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-25T19:43:22.169 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-25T19:43:22.630 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-25T19:43:22.695 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:22.721 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:22.721 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:43:22.721 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T19:43:22.721 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T19:43:22.722 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T19:43:22.722 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:23.974 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:23.974 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:23.974 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-25T19:43:23.974 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:43:23.974 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:23.975 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:43:24.247 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-nbd 2024-04-25T19:43:24.794 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:07 ago on Thu 25 Apr 2024 07:41:17 PM UTC. 2024-04-25T19:43:24.919 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-25T19:43:24.920 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:43:24.920 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-04-25T19:43:24.920 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:43:24.920 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-25T19:43:24.920 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T19:43:24.920 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:24.920 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-25T19:43:24.920 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-25T19:43:24.920 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-04-25T19:43:24.920 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:24.921 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 169 k 2024-04-25T19:43:24.921 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 540 k 2024-04-25T19:43:24.921 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-25T19:43:25.162 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-nbd-19.0.0-3291.g994e2f20.el9.x86_64.rpm 706 kB/s | 169 kB 00:00 2024-04-25T19:43:25.162 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:43:25.162 INFO:teuthology.orchestra.run.smithi029.stdout:Total 701 kB/s | 169 kB 00:00 2024-04-25T19:43:25.163 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-25T19:43:25.171 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-25T19:43:25.171 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-25T19:43:25.238 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-25T19:43:25.239 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-25T19:43:25.397 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-25T19:43:25.504 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:25.918 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:26.157 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 1/1 2024-04-25T19:43:26.157 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:26.157 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-25T19:43:26.157 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:43:26.157 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:26.157 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-25T19:43:26.430 DEBUG:teuthology.parallel:result is None 2024-04-25T19:43:26.430 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T19:43:26.571 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T19:43:26.594 INFO:teuthology.orchestra.run.smithi029.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T19:43:26.594 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T19:43:26.594 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T19:43:26.597 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T19:43:26.812 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T19:43:26.836 INFO:teuthology.orchestra.run.smithi102.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T19:43:26.837 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T19:43:26.837 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T19:43:26.839 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T19:43:26.839 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-25T19:43:26.839 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T19:43:26.867 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T19:43:26.867 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T19:43:26.903 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T19:43:26.903 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-25T19:43:26.903 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T19:43:26.932 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T19:43:26.998 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T19:43:26.998 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T19:43:27.024 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T19:43:27.089 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T19:43:27.089 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-25T19:43:27.089 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T19:43:27.114 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T19:43:27.178 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T19:43:27.179 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T19:43:27.203 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T19:43:27.268 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T19:43:27.268 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-25T19:43:27.268 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T19:43:27.293 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T19:43:27.358 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T19:43:27.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T19:43:27.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T19:43:27.446 INFO:teuthology.run_tasks:Running task ceph... 2024-04-25T19:43:27.536 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-25T19:43:27.536 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /var/log/ceph 2024-04-25T19:43:27.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 777 /var/log/ceph 2024-04-25T19:43:27.562 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-25T19:43:27.562 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T19:43:27.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T19:43:27.627 INFO:tasks.ceph:Creating extra log directories... 2024-04-25T19:43:27.627 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T19:43:27.669 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T19:43:27.698 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-25T19:43:27.698 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-25T19:43:27.698 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-25T19:43:27.698 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-25T19:43:27.698 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-25T19:43:27.747 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-25T19:43:27.858 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-25T19:43:27.883 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T19:43:27.883 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-25T19:43:27.898 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-25T19:43:28.001 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-25T19:43:28.049 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-25T19:43:28.049 INFO:tasks.ceph:config {'conf': {'client': {'debug client': 20, 'debug mgrc': 20, 'debug ms': 1}, 'global': {'mon election default strategy': 1}, 'mds': {'debug mgrc': 20}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1, 'mgr ttl cache expire seconds': 0}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug mgrc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_', 'replacing it with standby', 'No standby daemons available', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-25T19:43:28.049 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-04-25_11:25:24-rados:mgr-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673402', 'branch': 'wip-pdonnell-testing-20240425.015853-debug', 'description': 'rados:mgr/{clusters/{2-node-mgr} debug/mgr distro/{centos_latest} mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-hybrid} tasks/{1-install 2-ceph 3-mgrmodules 4-units/workunits}}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7673402', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'pdonnell-2024-04-25_11:25:24-rados:mgr-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 2, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20240425.015853-debug'}, 'ceph': {'conf': {'client': {'debug client': 20, 'debug mgrc': 20, 'debug ms': 1}, 'global': {'mon election default strategy': 1}, 'mds': {'debug mgrc': 20}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1, 'mgr ttl cache expire seconds': 0}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug mgrc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20240425.015853-debug', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 85, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'], ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1']], 'seed': 5640, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'sleep_before_teardown': 0, 'suite': 'rados:mgr', 'suite_branch': 'wip-pdonnell-testing-20240425.015853-debug', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'targets': {'smithi029.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFTQf+YLUeZx3Qzl6VT2y+2b6pyxkwYpUb7FnJerV+A9TCdXrtKOhp/EXR8H9KBCwKURpDh6lM4swsNxSA9OY/M=', 'smithi102.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJWFpWP0AIgZgBUgvEzasqXppOmczEePF8QeyxB5Dw5gLBggdtMDyxT/pr3HSP7+6836gvoDMHiZMJL5uaLsFn4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'log-ignorelist': ['overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_', 'replacing it with standby', 'No standby daemons available', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'wait-for-scrub': False, 'conf': {'client': {'debug client': 20, 'debug mgrc': 20, 'debug ms': 1}, 'global': {'mon election default strategy': 1}, 'mds': {'debug mgrc': 20}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1, 'mgr ttl cache expire seconds': 0}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug mgrc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'workunit': {'clients': {'client.0': ['mgr/test_localpool.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-25_11:25:24', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-25T19:43:28.050 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T19:43:28.053 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T19:43:28.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T19:43:28.109 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T19:43:28.160 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-25T19:43:28.160 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T19:43:28.186 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T19:43:28.187 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2024-04-25T19:43:28.242 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T19:43:28.242 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T19:43:28.242 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-25T19:43:28.242 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T19:43:28.242 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T19:43:28.242 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-04-25 19:43:22.967392827 +0000 2024-04-25T19:43:28.242 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-25 19:40:38.874070530 +0000 2024-04-25T19:43:28.242 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-25 19:40:38.874070530 +0000 2024-04-25T19:43:28.242 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-25 19:40:38.874070530 +0000 2024-04-25T19:43:28.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T19:43:28.304 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-04-25T19:43:28.304 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-04-25T19:43:28.304 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000100495 s, 5.1 MB/s 2024-04-25T19:43:28.305 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T19:43:28.362 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2024-04-25T19:43:28.416 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T19:43:28.417 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T19:43:28.417 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-25T19:43:28.417 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T19:43:28.417 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T19:43:28.417 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-04-25 19:43:22.968392811 +0000 2024-04-25T19:43:28.417 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-25 19:40:39.212064947 +0000 2024-04-25T19:43:28.417 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-25 19:40:39.212064947 +0000 2024-04-25T19:43:28.417 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-25 19:40:39.212064947 +0000 2024-04-25T19:43:28.417 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T19:43:28.479 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-04-25T19:43:28.479 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-04-25T19:43:28.479 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.4673e-05 s, 5.4 MB/s 2024-04-25T19:43:28.480 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T19:43:28.537 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2024-04-25T19:43:28.592 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T19:43:28.592 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T19:43:28.592 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-25T19:43:28.592 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T19:43:28.592 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T19:43:28.592 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-04-25 19:43:22.968392811 +0000 2024-04-25T19:43:28.592 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-25 19:40:39.560059200 +0000 2024-04-25T19:43:28.592 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-25 19:40:39.560059200 +0000 2024-04-25T19:43:28.592 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-25 19:40:39.560059200 +0000 2024-04-25T19:43:28.593 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T19:43:28.655 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-04-25T19:43:28.655 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-04-25T19:43:28.655 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 8.9933e-05 s, 5.7 MB/s 2024-04-25T19:43:28.656 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T19:43:28.712 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2024-04-25T19:43:28.767 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T19:43:28.767 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T19:43:28.767 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-25T19:43:28.767 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T19:43:28.767 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T19:43:28.767 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-04-25 19:43:22.968392811 +0000 2024-04-25T19:43:28.767 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-25 19:40:39.901053568 +0000 2024-04-25T19:43:28.767 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-25 19:40:39.901053568 +0000 2024-04-25T19:43:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-25 19:40:39.901053568 +0000 2024-04-25T19:43:28.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T19:43:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-04-25T19:43:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-04-25T19:43:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.774e-05 s, 5.2 MB/s 2024-04-25T19:43:28.831 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T19:43:28.887 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1'} 2024-04-25T19:43:28.887 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T19:43:28.887 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T19:43:28.903 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T19:43:28.904 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_1 2024-04-25T19:43:28.958 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T19:43:28.958 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T19:43:28.958 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-25T19:43:28.959 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T19:43:28.959 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T19:43:28.959 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-04-25 19:43:07.203881452 +0000 2024-04-25T19:43:28.959 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-04-25 19:40:32.113108499 +0000 2024-04-25T19:43:28.959 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-04-25 19:40:32.113108499 +0000 2024-04-25T19:43:28.959 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-04-25 19:40:32.113108499 +0000 2024-04-25T19:43:28.959 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T19:43:29.020 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-04-25T19:43:29.020 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-04-25T19:43:29.020 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000242954 s, 2.1 MB/s 2024-04-25T19:43:29.021 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T19:43:29.077 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_2 2024-04-25T19:43:29.132 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T19:43:29.132 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T19:43:29.132 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-25T19:43:29.132 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T19:43:29.133 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T19:43:29.133 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-04-25 19:43:07.203881452 +0000 2024-04-25T19:43:29.133 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-04-25 19:40:32.458100979 +0000 2024-04-25T19:43:29.133 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-04-25 19:40:32.458100979 +0000 2024-04-25T19:43:29.133 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-04-25 19:40:32.458100979 +0000 2024-04-25T19:43:29.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T19:43:29.194 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-04-25T19:43:29.194 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-04-25T19:43:29.194 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000232341 s, 2.2 MB/s 2024-04-25T19:43:29.195 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T19:43:29.251 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_3 2024-04-25T19:43:29.306 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T19:43:29.306 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T19:43:29.306 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-25T19:43:29.306 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T19:43:29.306 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T19:43:29.306 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-04-25 19:43:07.203881452 +0000 2024-04-25T19:43:29.306 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-04-25 19:40:32.805093415 +0000 2024-04-25T19:43:29.306 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-04-25 19:40:32.805093415 +0000 2024-04-25T19:43:29.307 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-04-25 19:40:32.805093415 +0000 2024-04-25T19:43:29.307 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T19:43:29.368 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-04-25T19:43:29.368 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-04-25T19:43:29.368 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000224819 s, 2.3 MB/s 2024-04-25T19:43:29.369 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T19:43:29.425 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_4 2024-04-25T19:43:29.479 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T19:43:29.479 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T19:43:29.479 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-25T19:43:29.479 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T19:43:29.479 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T19:43:29.479 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-04-25 19:43:07.204881432 +0000 2024-04-25T19:43:29.479 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-04-25 19:40:33.144086026 +0000 2024-04-25T19:43:29.480 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-04-25 19:40:33.144086026 +0000 2024-04-25T19:43:29.480 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-04-25 19:40:33.144086026 +0000 2024-04-25T19:43:29.480 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T19:43:29.541 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-04-25T19:43:29.541 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-04-25T19:43:29.541 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000207149 s, 2.5 MB/s 2024-04-25T19:43:29.542 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T19:43:29.598 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2024-04-25T19:43:29.598 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1'}, Remote(name='ubuntu@smithi102.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'}} 2024-04-25T19:43:29.598 INFO:tasks.ceph:Generating config... 2024-04-25T19:43:29.599 INFO:tasks.ceph:[client] debug client = 20 2024-04-25T19:43:29.600 INFO:tasks.ceph:[client] debug mgrc = 20 2024-04-25T19:43:29.600 INFO:tasks.ceph:[client] debug ms = 1 2024-04-25T19:43:29.600 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-04-25T19:43:29.600 INFO:tasks.ceph:[mds] debug mgrc = 20 2024-04-25T19:43:29.600 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-25T19:43:29.600 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-25T19:43:29.600 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-25T19:43:29.600 INFO:tasks.ceph:[mgr] mgr ttl cache expire seconds = 0 2024-04-25T19:43:29.600 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-25T19:43:29.600 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-25T19:43:29.600 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-25T19:43:29.600 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-04-25T19:43:29.600 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-04-25T19:43:29.600 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-25T19:43:29.600 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-25T19:43:29.601 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-25T19:43:29.601 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-25T19:43:29.601 INFO:tasks.ceph:[osd] debug mgrc = 20 2024-04-25T19:43:29.601 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-25T19:43:29.601 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-25T19:43:29.601 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-25T19:43:29.601 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-25T19:43:29.601 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-25T19:43:29.601 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-25T19:43:29.601 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-25T19:43:29.601 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-25T19:43:29.601 INFO:tasks.ceph:Setting up mon.a... 2024-04-25T19:43:29.602 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-25T19:43:29.661 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.keyring 2024-04-25T19:43:29.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-25T19:43:29.718 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T19:43:29.742 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.29'), ('mon.c', '[v2:172.21.15.29:3301,v1:172.21.15.29:6790]'), ('mon.b', '172.21.15.102')] 2024-04-25T19:43:29.743 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.29,[v2:172.21.15.29:3301,v1:172.21.15.29:6790],172.21.15.102', 'mon election default strategy': 1}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug mgrc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'mgr ttl cache expire seconds': 0}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug client': 20, 'debug mgrc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds': {'debug mgrc': 20}} 2024-04-25T19:43:29.743 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-25T19:43:29.744 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T19:43:29.799 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.29 --addv c '[v2:172.21.15.29:3301,v1:172.21.15.29:6790]' --add b 172.21.15.102 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:43:29.894 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:43:29.895 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: generated fsid 1b3c39b5-e787-409e-b0bd-f4f491c89d72 2024-04-25T19:43:29.895 INFO:teuthology.orchestra.run.smithi029.stdout:setting min_mon_release = quincy 2024-04-25T19:43:29.895 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 0 2024-04-25T19:43:29.895 INFO:teuthology.orchestra.run.smithi029.stdout:fsid 1b3c39b5-e787-409e-b0bd-f4f491c89d72 2024-04-25T19:43:29.895 INFO:teuthology.orchestra.run.smithi029.stdout:last_changed 2024-04-25T19:43:29.894972+0000 2024-04-25T19:43:29.895 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-04-25T19:43:29.894972+0000 2024-04-25T19:43:29.895 INFO:teuthology.orchestra.run.smithi029.stdout:min_mon_release 17 (quincy) 2024-04-25T19:43:29.895 INFO:teuthology.orchestra.run.smithi029.stdout:election_strategy: 1 2024-04-25T19:43:29.895 INFO:teuthology.orchestra.run.smithi029.stdout:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-04-25T19:43:29.895 INFO:teuthology.orchestra.run.smithi029.stdout:1: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.b 2024-04-25T19:43:29.895 INFO:teuthology.orchestra.run.smithi029.stdout:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2024-04-25T19:43:29.895 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-25T19:43:29.896 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T19:43:29.911 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 1b3c39b5-e787-409e-b0bd-f4f491c89d72... 2024-04-25T19:43:29.912 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T19:43:29.954 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T19:43:29.993 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2024-04-25T19:43:29.994 INFO:teuthology.orchestra.run.smithi029.stdout: chdir = "" 2024-04-25T19:43:29.994 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T19:43:29.994 INFO:teuthology.orchestra.run.smithi029.stdout: auth supported = cephx 2024-04-25T19:43:29.994 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.994 INFO:teuthology.orchestra.run.smithi029.stdout: filestore xattr use omap = true 2024-04-25T19:43:29.994 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.994 INFO:teuthology.orchestra.run.smithi029.stdout: mon clock drift allowed = 1.000 2024-04-25T19:43:29.994 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.994 INFO:teuthology.orchestra.run.smithi029.stdout: osd crush chooseleaf type = 0 2024-04-25T19:43:29.994 INFO:teuthology.orchestra.run.smithi029.stdout: auth debug = true 2024-04-25T19:43:29.994 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.994 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on old message = true 2024-04-25T19:43:29.994 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on bug = true 2024-04-25T19:43:29.995 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.995 INFO:teuthology.orchestra.run.smithi029.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T19:43:29.995 INFO:teuthology.orchestra.run.smithi029.stdout: mon pg warn max object skew = 0 2024-04-25T19:43:29.995 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.995 INFO:teuthology.orchestra.run.smithi029.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T19:43:29.995 INFO:teuthology.orchestra.run.smithi029.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T19:43:29.995 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.995 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default size = 2 2024-04-25T19:43:29.995 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.995 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow primary affinity = true 2024-04-25T19:43:29.995 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow pg remap = true 2024-04-25T19:43:29.995 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on legacy crush tunables = false 2024-04-25T19:43:29.995 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on crush straw calc version zero = false 2024-04-25T19:43:29.995 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on no sortbitwise = false 2024-04-25T19:43:29.995 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on osd down out interval zero = false 2024-04-25T19:43:29.996 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on too few osds = false 2024-04-25T19:43:29.996 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T19:43:29.996 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T19:43:29.996 INFO:teuthology.orchestra.run.smithi029.stdout: mon_allow_pool_size_one = true 2024-04-25T19:43:29.996 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.996 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T19:43:29.996 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.996 INFO:teuthology.orchestra.run.smithi029.stdout: osd default data pool replay window = 5 2024-04-25T19:43:29.996 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.996 INFO:teuthology.orchestra.run.smithi029.stdout: mon allow pool delete = true 2024-04-25T19:43:29.996 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.996 INFO:teuthology.orchestra.run.smithi029.stdout: mon cluster log file level = debug 2024-04-25T19:43:29.996 INFO:teuthology.orchestra.run.smithi029.stdout: debug asserts on shutdown = true 2024-04-25T19:43:29.996 INFO:teuthology.orchestra.run.smithi029.stdout: mon health detail to clog = false 2024-04-25T19:43:29.996 INFO:teuthology.orchestra.run.smithi029.stdout: mon host = "172.21.15.29,[v2:172.21.15.29:3301,v1:172.21.15.29:6790],172.21.15.102" 2024-04-25T19:43:29.997 INFO:teuthology.orchestra.run.smithi029.stdout: mon election default strategy = 1 2024-04-25T19:43:29.997 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = 1b3c39b5-e787-409e-b0bd-f4f491c89d72 2024-04-25T19:43:29.997 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.997 INFO:teuthology.orchestra.run.smithi029.stdout:[osd] 2024-04-25T19:43:29.997 INFO:teuthology.orchestra.run.smithi029.stdout: osd journal size = 100 2024-04-25T19:43:29.997 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.997 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub load threshold = 5.0 2024-04-25T19:43:29.997 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub max interval = 600 2024-04-25T19:43:29.997 INFO:teuthology.orchestra.run.smithi029.stdout: osd mclock profile = high_recovery_ops 2024-04-25T19:43:29.997 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.997 INFO:teuthology.orchestra.run.smithi029.stdout: osd recover clone overlap = true 2024-04-25T19:43:29.997 INFO:teuthology.orchestra.run.smithi029.stdout: osd recovery max chunk = 1048576 2024-04-25T19:43:29.997 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.997 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug shutdown = true 2024-04-25T19:43:29.997 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug op order = true 2024-04-25T19:43:29.998 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug verify stray on activate = true 2024-04-25T19:43:29.998 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug trim objects = true 2024-04-25T19:43:29.998 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.998 INFO:teuthology.orchestra.run.smithi029.stdout: osd open classes on start = true 2024-04-25T19:43:29.998 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug pg log writeout = true 2024-04-25T19:43:29.998 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.998 INFO:teuthology.orchestra.run.smithi029.stdout: osd deep scrub update digest min age = 30 2024-04-25T19:43:29.998 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.998 INFO:teuthology.orchestra.run.smithi029.stdout: osd map max advance = 10 2024-04-25T19:43:29.998 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.998 INFO:teuthology.orchestra.run.smithi029.stdout: journal zero on create = true 2024-04-25T19:43:29.998 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.998 INFO:teuthology.orchestra.run.smithi029.stdout: filestore ondisk finisher threads = 3 2024-04-25T19:43:29.998 INFO:teuthology.orchestra.run.smithi029.stdout: filestore apply finisher threads = 3 2024-04-25T19:43:29.999 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:29.999 INFO:teuthology.orchestra.run.smithi029.stdout: bdev debug aio = true 2024-04-25T19:43:29.999 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug misdirected ops = true 2024-04-25T19:43:29.999 INFO:teuthology.orchestra.run.smithi029.stdout: bluefs allocator = hybrid 2024-04-25T19:43:29.999 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore allocator = hybrid 2024-04-25T19:43:29.999 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore block size = 96636764160 2024-04-25T19:43:29.999 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore fsck on mount = True 2024-04-25T19:43:29.999 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluefs = 20 2024-04-25T19:43:29.999 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluestore = 20 2024-04-25T19:43:29.999 INFO:teuthology.orchestra.run.smithi029.stdout: debug mgrc = 20 2024-04-25T19:43:29.999 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2024-04-25T19:43:29.999 INFO:teuthology.orchestra.run.smithi029.stdout: debug osd = 20 2024-04-25T19:43:29.999 INFO:teuthology.orchestra.run.smithi029.stdout: debug rocksdb = 10 2024-04-25T19:43:29.999 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T19:43:29.999 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd full ratio = 0.9 2024-04-25T19:43:30.000 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd nearfull ratio = 0.8 2024-04-25T19:43:30.000 INFO:teuthology.orchestra.run.smithi029.stdout: osd failsafe full ratio = 0.95 2024-04-25T19:43:30.000 INFO:teuthology.orchestra.run.smithi029.stdout: osd objectstore = bluestore 2024-04-25T19:43:30.000 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:30.000 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr] 2024-04-25T19:43:30.000 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2024-04-25T19:43:30.000 INFO:teuthology.orchestra.run.smithi029.stdout: debug mgr = 20 2024-04-25T19:43:30.000 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2024-04-25T19:43:30.000 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2024-04-25T19:43:30.000 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2024-04-25T19:43:30.000 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2024-04-25T19:43:30.000 INFO:teuthology.orchestra.run.smithi029.stdout: mgr/telemetry/nag = false 2024-04-25T19:43:30.000 INFO:teuthology.orchestra.run.smithi029.stdout: debug client = 20 2024-04-25T19:43:30.000 INFO:teuthology.orchestra.run.smithi029.stdout: mgr ttl cache expire seconds = 0 2024-04-25T19:43:30.000 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:30.001 INFO:teuthology.orchestra.run.smithi029.stdout:[mon] 2024-04-25T19:43:30.001 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2024-04-25T19:43:30.001 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2024-04-25T19:43:30.001 INFO:teuthology.orchestra.run.smithi029.stdout: debug paxos = 20 2024-04-25T19:43:30.001 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2024-04-25T19:43:30.001 INFO:teuthology.orchestra.run.smithi029.stdout: mon data avail warn = 5 2024-04-25T19:43:30.001 INFO:teuthology.orchestra.run.smithi029.stdout: mon mgr mkfs grace = 240 2024-04-25T19:43:30.001 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2024-04-25T19:43:30.001 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd reporter subtree level = osd 2024-04-25T19:43:30.001 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd prime pg temp = true 2024-04-25T19:43:30.001 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2024-04-25T19:43:30.001 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:30.001 INFO:teuthology.orchestra.run.smithi029.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T19:43:30.001 INFO:teuthology.orchestra.run.smithi029.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T19:43:30.001 INFO:teuthology.orchestra.run.smithi029.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T19:43:30.002 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:30.002 INFO:teuthology.orchestra.run.smithi029.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T19:43:30.002 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T19:43:30.002 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T19:43:30.002 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:30.002 INFO:teuthology.orchestra.run.smithi029.stdout: # 1m isn't quite enough 2024-04-25T19:43:30.002 INFO:teuthology.orchestra.run.smithi029.stdout: mon_down_mkfs_grace = 2m 2024-04-25T19:43:30.002 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:30.002 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_filestore_osds = false 2024-04-25T19:43:30.002 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:30.002 INFO:teuthology.orchestra.run.smithi029.stdout:[client] 2024-04-25T19:43:30.002 INFO:teuthology.orchestra.run.smithi029.stdout: rgw cache enabled = true 2024-04-25T19:43:30.002 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable ops log = true 2024-04-25T19:43:30.002 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable usage log = true 2024-04-25T19:43:30.002 INFO:teuthology.orchestra.run.smithi029.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T19:43:30.003 INFO:teuthology.orchestra.run.smithi029.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T19:43:30.003 INFO:teuthology.orchestra.run.smithi029.stdout: debug client = 20 2024-04-25T19:43:30.003 INFO:teuthology.orchestra.run.smithi029.stdout: debug mgrc = 20 2024-04-25T19:43:30.003 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2024-04-25T19:43:30.003 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.a] 2024-04-25T19:43:30.003 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.c] 2024-04-25T19:43:30.003 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.b] 2024-04-25T19:43:30.003 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.a] 2024-04-25T19:43:30.003 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.c] 2024-04-25T19:43:30.003 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.b] 2024-04-25T19:43:30.003 INFO:teuthology.orchestra.run.smithi029.stdout:[mds] 2024-04-25T19:43:30.003 INFO:teuthology.orchestra.run.smithi029.stdout: debug mgrc = 20 2024-04-25T19:43:30.006 INFO:teuthology.orchestra.run.smithi102.stdout:[global] 2024-04-25T19:43:30.006 INFO:teuthology.orchestra.run.smithi102.stdout: chdir = "" 2024-04-25T19:43:30.006 INFO:teuthology.orchestra.run.smithi102.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T19:43:30.006 INFO:teuthology.orchestra.run.smithi102.stdout: auth supported = cephx 2024-04-25T19:43:30.007 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.007 INFO:teuthology.orchestra.run.smithi102.stdout: filestore xattr use omap = true 2024-04-25T19:43:30.007 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.007 INFO:teuthology.orchestra.run.smithi102.stdout: mon clock drift allowed = 1.000 2024-04-25T19:43:30.007 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.007 INFO:teuthology.orchestra.run.smithi102.stdout: osd crush chooseleaf type = 0 2024-04-25T19:43:30.007 INFO:teuthology.orchestra.run.smithi102.stdout: auth debug = true 2024-04-25T19:43:30.007 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.007 INFO:teuthology.orchestra.run.smithi102.stdout: ms die on old message = true 2024-04-25T19:43:30.007 INFO:teuthology.orchestra.run.smithi102.stdout: ms die on bug = true 2024-04-25T19:43:30.007 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.007 INFO:teuthology.orchestra.run.smithi102.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T19:43:30.007 INFO:teuthology.orchestra.run.smithi102.stdout: mon pg warn max object skew = 0 2024-04-25T19:43:30.007 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.007 INFO:teuthology.orchestra.run.smithi102.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T19:43:30.008 INFO:teuthology.orchestra.run.smithi102.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T19:43:30.008 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.008 INFO:teuthology.orchestra.run.smithi102.stdout: osd pool default size = 2 2024-04-25T19:43:30.008 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.008 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd allow primary affinity = true 2024-04-25T19:43:30.008 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd allow pg remap = true 2024-04-25T19:43:30.008 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on legacy crush tunables = false 2024-04-25T19:43:30.008 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on crush straw calc version zero = false 2024-04-25T19:43:30.008 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on no sortbitwise = false 2024-04-25T19:43:30.008 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on osd down out interval zero = false 2024-04-25T19:43:30.008 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on too few osds = false 2024-04-25T19:43:30.008 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T19:43:30.008 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T19:43:30.008 INFO:teuthology.orchestra.run.smithi102.stdout: mon_allow_pool_size_one = true 2024-04-25T19:43:30.008 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.009 INFO:teuthology.orchestra.run.smithi102.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T19:43:30.009 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.009 INFO:teuthology.orchestra.run.smithi102.stdout: osd default data pool replay window = 5 2024-04-25T19:43:30.009 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.009 INFO:teuthology.orchestra.run.smithi102.stdout: mon allow pool delete = true 2024-04-25T19:43:30.009 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.009 INFO:teuthology.orchestra.run.smithi102.stdout: mon cluster log file level = debug 2024-04-25T19:43:30.009 INFO:teuthology.orchestra.run.smithi102.stdout: debug asserts on shutdown = true 2024-04-25T19:43:30.009 INFO:teuthology.orchestra.run.smithi102.stdout: mon health detail to clog = false 2024-04-25T19:43:30.009 INFO:teuthology.orchestra.run.smithi102.stdout: mon host = "172.21.15.29,[v2:172.21.15.29:3301,v1:172.21.15.29:6790],172.21.15.102" 2024-04-25T19:43:30.009 INFO:teuthology.orchestra.run.smithi102.stdout: mon election default strategy = 1 2024-04-25T19:43:30.009 INFO:teuthology.orchestra.run.smithi102.stdout: fsid = 1b3c39b5-e787-409e-b0bd-f4f491c89d72 2024-04-25T19:43:30.009 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.009 INFO:teuthology.orchestra.run.smithi102.stdout:[osd] 2024-04-25T19:43:30.009 INFO:teuthology.orchestra.run.smithi102.stdout: osd journal size = 100 2024-04-25T19:43:30.010 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.010 INFO:teuthology.orchestra.run.smithi102.stdout: osd scrub load threshold = 5.0 2024-04-25T19:43:30.010 INFO:teuthology.orchestra.run.smithi102.stdout: osd scrub max interval = 600 2024-04-25T19:43:30.010 INFO:teuthology.orchestra.run.smithi102.stdout: osd mclock profile = high_recovery_ops 2024-04-25T19:43:30.010 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.010 INFO:teuthology.orchestra.run.smithi102.stdout: osd recover clone overlap = true 2024-04-25T19:43:30.010 INFO:teuthology.orchestra.run.smithi102.stdout: osd recovery max chunk = 1048576 2024-04-25T19:43:30.010 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.010 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug shutdown = true 2024-04-25T19:43:30.010 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug op order = true 2024-04-25T19:43:30.010 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug verify stray on activate = true 2024-04-25T19:43:30.010 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug trim objects = true 2024-04-25T19:43:30.010 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.010 INFO:teuthology.orchestra.run.smithi102.stdout: osd open classes on start = true 2024-04-25T19:43:30.011 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug pg log writeout = true 2024-04-25T19:43:30.011 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.011 INFO:teuthology.orchestra.run.smithi102.stdout: osd deep scrub update digest min age = 30 2024-04-25T19:43:30.011 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.011 INFO:teuthology.orchestra.run.smithi102.stdout: osd map max advance = 10 2024-04-25T19:43:30.011 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.011 INFO:teuthology.orchestra.run.smithi102.stdout: journal zero on create = true 2024-04-25T19:43:30.011 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.011 INFO:teuthology.orchestra.run.smithi102.stdout: filestore ondisk finisher threads = 3 2024-04-25T19:43:30.011 INFO:teuthology.orchestra.run.smithi102.stdout: filestore apply finisher threads = 3 2024-04-25T19:43:30.011 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.011 INFO:teuthology.orchestra.run.smithi102.stdout: bdev debug aio = true 2024-04-25T19:43:30.011 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug misdirected ops = true 2024-04-25T19:43:30.011 INFO:teuthology.orchestra.run.smithi102.stdout: bluefs allocator = hybrid 2024-04-25T19:43:30.011 INFO:teuthology.orchestra.run.smithi102.stdout: bluestore allocator = hybrid 2024-04-25T19:43:30.011 INFO:teuthology.orchestra.run.smithi102.stdout: bluestore block size = 96636764160 2024-04-25T19:43:30.012 INFO:teuthology.orchestra.run.smithi102.stdout: bluestore fsck on mount = True 2024-04-25T19:43:30.012 INFO:teuthology.orchestra.run.smithi102.stdout: debug bluefs = 20 2024-04-25T19:43:30.012 INFO:teuthology.orchestra.run.smithi102.stdout: debug bluestore = 20 2024-04-25T19:43:30.012 INFO:teuthology.orchestra.run.smithi102.stdout: debug mgrc = 20 2024-04-25T19:43:30.012 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-04-25T19:43:30.012 INFO:teuthology.orchestra.run.smithi102.stdout: debug osd = 20 2024-04-25T19:43:30.012 INFO:teuthology.orchestra.run.smithi102.stdout: debug rocksdb = 10 2024-04-25T19:43:30.012 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T19:43:30.012 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd full ratio = 0.9 2024-04-25T19:43:30.012 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd nearfull ratio = 0.8 2024-04-25T19:43:30.012 INFO:teuthology.orchestra.run.smithi102.stdout: osd failsafe full ratio = 0.95 2024-04-25T19:43:30.012 INFO:teuthology.orchestra.run.smithi102.stdout: osd objectstore = bluestore 2024-04-25T19:43:30.012 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.012 INFO:teuthology.orchestra.run.smithi102.stdout:[mgr] 2024-04-25T19:43:30.012 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-04-25T19:43:30.013 INFO:teuthology.orchestra.run.smithi102.stdout: debug mgr = 20 2024-04-25T19:43:30.013 INFO:teuthology.orchestra.run.smithi102.stdout: debug mon = 20 2024-04-25T19:43:30.013 INFO:teuthology.orchestra.run.smithi102.stdout: debug auth = 20 2024-04-25T19:43:30.013 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min pgs per osd = 4 2024-04-25T19:43:30.013 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min bytes per osd = 10 2024-04-25T19:43:30.013 INFO:teuthology.orchestra.run.smithi102.stdout: mgr/telemetry/nag = false 2024-04-25T19:43:30.013 INFO:teuthology.orchestra.run.smithi102.stdout: debug client = 20 2024-04-25T19:43:30.013 INFO:teuthology.orchestra.run.smithi102.stdout: mgr ttl cache expire seconds = 0 2024-04-25T19:43:30.013 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.013 INFO:teuthology.orchestra.run.smithi102.stdout:[mon] 2024-04-25T19:43:30.013 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-04-25T19:43:30.013 INFO:teuthology.orchestra.run.smithi102.stdout: debug mon = 20 2024-04-25T19:43:30.013 INFO:teuthology.orchestra.run.smithi102.stdout: debug paxos = 20 2024-04-25T19:43:30.013 INFO:teuthology.orchestra.run.smithi102.stdout: debug auth = 20 2024-04-25T19:43:30.013 INFO:teuthology.orchestra.run.smithi102.stdout: mon data avail warn = 5 2024-04-25T19:43:30.014 INFO:teuthology.orchestra.run.smithi102.stdout: mon mgr mkfs grace = 240 2024-04-25T19:43:30.014 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min pgs per osd = 4 2024-04-25T19:43:30.014 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd reporter subtree level = osd 2024-04-25T19:43:30.014 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd prime pg temp = true 2024-04-25T19:43:30.014 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min bytes per osd = 10 2024-04-25T19:43:30.014 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.014 INFO:teuthology.orchestra.run.smithi102.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T19:43:30.014 INFO:teuthology.orchestra.run.smithi102.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T19:43:30.014 INFO:teuthology.orchestra.run.smithi102.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T19:43:30.014 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.014 INFO:teuthology.orchestra.run.smithi102.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T19:43:30.014 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T19:43:30.014 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T19:43:30.014 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.014 INFO:teuthology.orchestra.run.smithi102.stdout: # 1m isn't quite enough 2024-04-25T19:43:30.015 INFO:teuthology.orchestra.run.smithi102.stdout: mon_down_mkfs_grace = 2m 2024-04-25T19:43:30.015 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.015 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_filestore_osds = false 2024-04-25T19:43:30.015 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T19:43:30.015 INFO:teuthology.orchestra.run.smithi102.stdout:[client] 2024-04-25T19:43:30.015 INFO:teuthology.orchestra.run.smithi102.stdout: rgw cache enabled = true 2024-04-25T19:43:30.015 INFO:teuthology.orchestra.run.smithi102.stdout: rgw enable ops log = true 2024-04-25T19:43:30.015 INFO:teuthology.orchestra.run.smithi102.stdout: rgw enable usage log = true 2024-04-25T19:43:30.015 INFO:teuthology.orchestra.run.smithi102.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T19:43:30.015 INFO:teuthology.orchestra.run.smithi102.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T19:43:30.015 INFO:teuthology.orchestra.run.smithi102.stdout: debug client = 20 2024-04-25T19:43:30.015 INFO:teuthology.orchestra.run.smithi102.stdout: debug mgrc = 20 2024-04-25T19:43:30.015 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-04-25T19:43:30.015 INFO:teuthology.orchestra.run.smithi102.stdout:[mon.a] 2024-04-25T19:43:30.015 INFO:teuthology.orchestra.run.smithi102.stdout:[mon.c] 2024-04-25T19:43:30.015 INFO:teuthology.orchestra.run.smithi102.stdout:[mon.b] 2024-04-25T19:43:30.016 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.a] 2024-04-25T19:43:30.016 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.c] 2024-04-25T19:43:30.016 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.b] 2024-04-25T19:43:30.016 INFO:teuthology.orchestra.run.smithi102.stdout:[mds] 2024-04-25T19:43:30.016 INFO:teuthology.orchestra.run.smithi102.stdout: debug mgrc = 20 2024-04-25T19:43:30.016 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-25T19:43:30.016 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-25T19:43:30.110 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-25T19:43:30.110 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-25T19:43:30.110 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-25T19:43:30.125 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-25T19:43:30.125 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-25T19:43:30.180 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi029.front.sepia.ceph.com 2024-04-25T19:43:30.180 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-25T19:43:30.180 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T19:43:30.180 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T19:43:30.253 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-25T19:43:30.253 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:43:30.309 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T19:43:30.309 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T19:43:30.309 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T19:43:30.309 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T19:43:30.341 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T19:43:30.341 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:43:30.396 INFO:tasks.ceph:Setting up mon nodes... 2024-04-25T19:43:30.396 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-25T19:43:30.396 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T19:43:30.455 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T19:43:30.458 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-25T19:43:30.518 INFO:teuthology.orchestra.run.smithi102.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-25T19:43:30.521 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mgr/ceph-z && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.z /var/lib/ceph/mgr/ceph-z/keyring 2024-04-25T19:43:30.578 INFO:teuthology.orchestra.run.smithi102.stdout:creating /var/lib/ceph/mgr/ceph-z/keyring 2024-04-25T19:43:30.581 INFO:tasks.ceph:Setting up mds nodes... 2024-04-25T19:43:30.581 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-25T19:43:30.641 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-25T19:43:30.644 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-25T19:43:30.668 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-25T19:43:30.766 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-25T19:43:30.769 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-25T19:43:30.793 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-25T19:43:30.851 INFO:teuthology.orchestra.run.smithi102.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-25T19:43:30.853 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-25T19:43:30.876 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-25T19:43:30.876 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T19:43:30.926 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-25T19:43:30.938 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-25T19:43:30.986 INFO:teuthology.orchestra.run.smithi102.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-25T19:43:30.999 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-25T19:43:30.999 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1'}, Remote(name='ubuntu@smithi102.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'}} 2024-04-25T19:43:30.999 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-25T19:43:31.025 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1'} 2024-04-25T19:43:31.025 INFO:tasks.ceph:role: osd.0 2024-04-25T19:43:31.025 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com 2024-04-25T19:43:31.025 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T19:43:31.089 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T19:43:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T19:43:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T19:43:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T19:43:31.089 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T19:43:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2024-04-25T19:43:31.089 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T19:43:31.089 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T19:43:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T19:43:31.090 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T19:43:31.122 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2024-04-25T19:43:31.123 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2024-04-25T19:43:31.124 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-25T19:43:31.192 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-25T19:43:31.259 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:43:31.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:31.359+0000 7fa7bd799640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-25T19:43:31.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:31.360+0000 7fa7bd799640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-25T19:43:31.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:31.360+0000 7fa7bd799640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T19:43:31.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:31.360+0000 7fa7bd799640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T19:43:31.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:31.360+0000 7fa7bd799640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T19:43:31.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:31.360+0000 7fa7bd799640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-25T19:43:34.229 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-25T19:43:34.259 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-25T19:43:34.286 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2024-04-25T19:43:34.286 INFO:tasks.ceph:role: osd.1 2024-04-25T19:43:34.286 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T19:43:34.287 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T19:43:34.361 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T19:43:34.361 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T19:43:34.361 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T19:43:34.361 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T19:43:34.362 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T19:43:34.362 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2024-04-25T19:43:34.362 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T19:43:34.362 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T19:43:34.362 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T19:43:34.362 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T19:43:34.872 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2024-04-25T19:43:34.874 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2024-04-25T19:43:34.874 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2024-04-25T19:43:35.002 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-25T19:43:35.031 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-25T19:43:35.095 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2024-04-25T19:43:35.095 INFO:tasks.ceph:role: osd.2 2024-04-25T19:43:35.095 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T19:43:35.096 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T19:43:35.167 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T19:43:35.167 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T19:43:35.167 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T19:43:35.167 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T19:43:35.167 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T19:43:35.167 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2024-04-25T19:43:35.167 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T19:43:35.168 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T19:43:35.168 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T19:43:35.168 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T19:43:35.624 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2024-04-25T19:43:35.625 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2024-04-25T19:43:35.626 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2 2024-04-25T19:43:35.730 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-25T19:43:35.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-25T19:43:35.822 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2024-04-25T19:43:35.822 INFO:tasks.ceph:role: osd.3 2024-04-25T19:43:35.822 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T19:43:35.822 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T19:43:35.893 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T19:43:35.893 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T19:43:35.893 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T19:43:35.893 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T19:43:35.893 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T19:43:35.893 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2024-04-25T19:43:35.893 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T19:43:35.893 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T19:43:35.893 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T19:43:35.894 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T19:43:36.377 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2024-04-25T19:43:36.379 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2024-04-25T19:43:36.379 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-3 2024-04-25T19:43:36.523 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-25T19:43:36.552 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:43:36.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:36.651+0000 7f32b9041640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-25T19:43:36.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:36.657+0000 7f32b9041640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-25T19:43:36.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:36.657+0000 7f32b9041640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T19:43:36.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:36.657+0000 7f32b9041640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T19:43:36.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:36.657+0000 7f32b9041640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T19:43:36.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:36.664+0000 7f32b9041640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-25T19:43:39.519 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-25T19:43:39.548 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:43:39.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:39.648+0000 7f4b3b621640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-25T19:43:39.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:39.648+0000 7f4b3b621640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-25T19:43:39.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:39.648+0000 7f4b3b621640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T19:43:39.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:39.648+0000 7f4b3b621640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T19:43:39.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:39.648+0000 7f4b3b621640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T19:43:39.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:39.649+0000 7f4b3b621640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-25T19:43:42.504 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-25T19:43:42.531 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:43:42.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:42.629+0000 7f5aff826640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-25T19:43:42.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:42.630+0000 7f5aff826640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-25T19:43:42.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:42.630+0000 7f5aff826640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T19:43:42.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:42.630+0000 7f5aff826640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T19:43:42.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:42.630+0000 7f5aff826640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T19:43:42.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:42.630+0000 7f5aff826640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-25T19:43:45.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-25T19:43:45.507 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-25T19:43:45.507 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-25T19:43:45.507 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-25T19:43:45.538 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-25T19:43:45.538 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-25T19:43:45.606 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-25T19:43:45.606 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-25T19:43:45.673 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-25T19:43:45.673 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-25T19:43:45.739 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T19:43:45.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-25T19:43:45.765 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T19:43:45.765 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/mgr/ceph-z/keyring of=/dev/stdout 2024-04-25T19:43:45.832 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T19:43:45.833 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-25T19:43:45.897 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T19:43:45.897 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-25T19:43:45.960 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T19:43:45.960 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-25T19:43:46.024 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T19:43:46.024 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-25T19:43:46.087 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-25T19:43:46.087 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-25T19:43:46.106 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T19:43:46.106 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-25T19:43:46.142 INFO:tasks.ceph:Adding keys to all mons... 2024-04-25T19:43:46.143 DEBUG:teuthology.orchestra.run.smithi029:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T19:43:46.148 DEBUG:teuthology.orchestra.run.smithi102:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T19:43:46.186 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr.x] 2024-04-25T19:43:46.186 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBisipmhpAmGxAAHHdkD/WpC41wJRLLuIO34A== 2024-04-25T19:43:46.186 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.a] 2024-04-25T19:43:46.186 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBisipmHMBAJhAAmLCxDWHmfhlwjp+zbh9H6g== 2024-04-25T19:43:46.186 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.c] 2024-04-25T19:43:46.186 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBisipmJ5W4LRAAwPLFX89L9Iu+7W4MS3DR2Q== 2024-04-25T19:43:46.186 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.0] 2024-04-25T19:43:46.186 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBjsipmcECIFRAA4PAZHQtYWtEl4VTEus3LjA== 2024-04-25T19:43:46.186 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr.y] 2024-04-25T19:43:46.186 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBisipmNZvjHhAASDOA1WD2/BV+PeJGTK+fRQ== 2024-04-25T19:43:46.186 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr.z] 2024-04-25T19:43:46.187 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBisipmlz17IhAAYyoGhykvt1Zk5KBYNN1psw== 2024-04-25T19:43:46.187 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.b] 2024-04-25T19:43:46.187 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBisipmdPS5MhAA5cX6/72Fa/i8OqvM7RcWZg== 2024-04-25T19:43:46.187 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.1] 2024-04-25T19:43:46.187 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBosipmwOrfJhAAFtUnB+EvMHov5WopCvdimA== 2024-04-25T19:43:46.187 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.2] 2024-04-25T19:43:46.187 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBrsipmvcSzJhAAu+ilMUGHOA0sQE9fBt0T4w== 2024-04-25T19:43:46.187 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.3] 2024-04-25T19:43:46.187 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBusipmBlOYJRAAc1dWrTS4NNw8ktyQ+ZPhMw== 2024-04-25T19:43:46.187 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2024-04-25T19:43:46.187 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBisipmLKA2NxAAI05zJTI1R0XPVVRlxoL+1w== 2024-04-25T19:43:46.187 INFO:teuthology.orchestra.run.smithi029.stdout:[client.1] 2024-04-25T19:43:46.187 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBisipmfzvROhAAeYsjxHt5DxxroLw+sRWGqQ== 2024-04-25T19:43:46.206 INFO:teuthology.orchestra.run.smithi102.stdout:[mgr.x] 2024-04-25T19:43:46.207 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBisipmhpAmGxAAHHdkD/WpC41wJRLLuIO34A== 2024-04-25T19:43:46.207 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.a] 2024-04-25T19:43:46.207 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBisipmHMBAJhAAmLCxDWHmfhlwjp+zbh9H6g== 2024-04-25T19:43:46.207 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.c] 2024-04-25T19:43:46.207 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBisipmJ5W4LRAAwPLFX89L9Iu+7W4MS3DR2Q== 2024-04-25T19:43:46.207 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.0] 2024-04-25T19:43:46.207 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBjsipmcECIFRAA4PAZHQtYWtEl4VTEus3LjA== 2024-04-25T19:43:46.207 INFO:teuthology.orchestra.run.smithi102.stdout:[mgr.y] 2024-04-25T19:43:46.207 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBisipmNZvjHhAASDOA1WD2/BV+PeJGTK+fRQ== 2024-04-25T19:43:46.207 INFO:teuthology.orchestra.run.smithi102.stdout:[mgr.z] 2024-04-25T19:43:46.207 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBisipmlz17IhAAYyoGhykvt1Zk5KBYNN1psw== 2024-04-25T19:43:46.207 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.b] 2024-04-25T19:43:46.207 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBisipmdPS5MhAA5cX6/72Fa/i8OqvM7RcWZg== 2024-04-25T19:43:46.207 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.1] 2024-04-25T19:43:46.208 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBosipmwOrfJhAAFtUnB+EvMHov5WopCvdimA== 2024-04-25T19:43:46.208 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.2] 2024-04-25T19:43:46.208 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBrsipmvcSzJhAAu+ilMUGHOA0sQE9fBt0T4w== 2024-04-25T19:43:46.208 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.3] 2024-04-25T19:43:46.208 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBusipmBlOYJRAAc1dWrTS4NNw8ktyQ+ZPhMw== 2024-04-25T19:43:46.208 INFO:teuthology.orchestra.run.smithi102.stdout:[client.0] 2024-04-25T19:43:46.208 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBisipmLKA2NxAAI05zJTI1R0XPVVRlxoL+1w== 2024-04-25T19:43:46.208 INFO:teuthology.orchestra.run.smithi102.stdout:[client.1] 2024-04-25T19:43:46.208 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBisipmfzvROhAAeYsjxHt5DxxroLw+sRWGqQ== 2024-04-25T19:43:46.208 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T19:43:46.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T19:43:46.305 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T19:43:46.307 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T19:43:46.371 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T19:43:46.374 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T19:43:46.435 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:43:46.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:43:46.496 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T19:43:46.499 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T19:43:46.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.z --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T19:43:46.559 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.z --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T19:43:46.619 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T19:43:46.621 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T19:43:46.682 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:43:46.684 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:43:46.742 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:43:46.745 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:43:46.803 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:43:46.805 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:43:46.864 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T19:43:46.867 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T19:43:46.928 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T19:43:46.930 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T19:43:46.991 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-25T19:43:46.991 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-25T19:43:47.015 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T19:43:47.120 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-25T19:43:47.144 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-25T19:43:47.208 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T19:43:47.314 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-25T19:43:47.338 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-25T19:43:47.363 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T19:43:47.467 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-25T19:43:47.490 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:43:47.493 DEBUG:teuthology.orchestra.run.smithi102:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:43:47.545 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-25T19:43:47.545 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-25T19:43:47.545 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-25T19:43:47.548 INFO:tasks.ceph.mon.a:Started 2024-04-25T19:43:47.548 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-25T19:43:47.548 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-25T19:43:47.550 INFO:tasks.ceph.mon.c:Started 2024-04-25T19:43:47.550 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-25T19:43:47.550 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-25T19:43:47.588 INFO:tasks.ceph.mon.b:Started 2024-04-25T19:43:47.588 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-25T19:43:47.588 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-25T19:43:47.588 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-25T19:43:47.590 INFO:tasks.ceph.mgr.x:Started 2024-04-25T19:43:47.591 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-25T19:43:47.591 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-25T19:43:47.593 INFO:tasks.ceph.mgr.y:Started 2024-04-25T19:43:47.593 INFO:tasks.ceph.mgr.z:Restarting daemon 2024-04-25T19:43:47.593 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i z 2024-04-25T19:43:47.595 INFO:tasks.ceph.mgr.z:Started 2024-04-25T19:43:47.595 INFO:tasks.ceph:Setting crush tunables to default 2024-04-25T19:43:47.596 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd crush tunables default 2024-04-25T19:43:47.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.711+0000 7fb47f586640 1 Processor -- start 2024-04-25T19:43:47.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.711+0000 7fb47f586640 1 -- start start 2024-04-25T19:43:47.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.712+0000 7fb47f586640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb4780ba100 0x7fb4780addb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:47.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.712+0000 7fb47f586640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb4780b70f0 0x7fb4780b7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:47.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.712+0000 7fb47f586640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb478136f40 0x7fb478137350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:47.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.712+0000 7fb47f586640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fb4780ae2f0 con 0x7fb478136f40 2024-04-25T19:43:47.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.712+0000 7fb47f586640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fb4780ba510 con 0x7fb4780b70f0 2024-04-25T19:43:47.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.712+0000 7fb47f586640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fb478137bd0 con 0x7fb4780ba100 2024-04-25T19:43:47.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.712+0000 7fb47d322640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb4780ba100 0x7fb4780addb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:47.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.712+0000 7fb47db23640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb478136f40 0x7fb478137350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:47.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.712+0000 7fb47d322640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb4780ba100 0x7fb4780addb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:43240/0 (socket says 172.21.15.29:43240) 2024-04-25T19:43:47.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.712+0000 7fb47db23640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb478136f40 0x7fb478137350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58166/0 (socket says 172.21.15.29:58166) 2024-04-25T19:43:47.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.712+0000 7fb47d322640 1 -- 172.21.15.29:0/2810775000 learned_addr learned my addr 172.21.15.29:0/2810775000 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:43:47.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.712+0000 7fb47d322640 1 -- 172.21.15.29:0/2810775000 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb4780ba100 msgr2=0x7fb4780addb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T19:43:47.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.713+0000 7fb47d322640 1 -- 172.21.15.29:0/2810775000 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb4780ba100 msgr2=0x7fb4780addb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T19:43:47.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.713+0000 7fb47db23640 1 -- 172.21.15.29:0/2810775000 >> v2:172.21.15.29:3300/0 conn(0x7fb478136f40 msgr2=0x7fb478137350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-25T19:43:47.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.713+0000 7fb47d322640 1 --2- 172.21.15.29:0/2810775000 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb4780ba100 0x7fb4780addb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T19:43:47.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.713+0000 7fb47db23640 1 -- 172.21.15.29:0/2810775000 >> v2:172.21.15.29:3300/0 conn(0x7fb478136f40 msgr2=0x7fb478137350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T19:43:47.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.713+0000 7fb47db23640 1 --2- 172.21.15.29:0/2810775000 >> v2:172.21.15.29:3300/0 conn(0x7fb478136f40 0x7fb478137350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T19:43:47.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.713+0000 7fb47cb21640 1 --2- 172.21.15.29:0/2810775000 >> v2:172.21.15.102:3300/0 conn(0x7fb4780b70f0 0x7fb4780b7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:47.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.713+0000 7fb47db23640 1 --2- 172.21.15.29:0/2810775000 >> v2:172.21.15.29:3300/0 conn(0x7fb478136f40 0x7fb478137350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T19:43:47.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.713+0000 7fb47d322640 1 --2- 172.21.15.29:0/2810775000 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb4780ba100 0x7fb4780addb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T19:43:47.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.713+0000 7fb47cb21640 1 -- 172.21.15.29:0/2810775000 >> v2:172.21.15.102:3300/0 conn(0x7fb4780b70f0 msgr2=0x7fb4780b7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T19:43:47.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.713+0000 7fb47cb21640 1 -- 172.21.15.29:0/2810775000 >> v2:172.21.15.102:3300/0 conn(0x7fb4780b70f0 msgr2=0x7fb4780b7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T19:43:47.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.713+0000 7fb47cb21640 1 --2- 172.21.15.29:0/2810775000 >> v2:172.21.15.102:3300/0 conn(0x7fb4780b70f0 0x7fb4780b7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T19:43:47.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.713+0000 7fb47cb21640 1 --2- 172.21.15.29:0/2810775000 >> v2:172.21.15.102:3300/0 conn(0x7fb4780b70f0 0x7fb4780b7500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T19:43:47.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.914+0000 7fb47db23640 1 --2- 172.21.15.29:0/2810775000 >> v2:172.21.15.29:3300/0 conn(0x7fb478136f40 0x7fb478137350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:47.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.914+0000 7fb47d322640 1 --2- 172.21.15.29:0/2810775000 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb4780ba100 0x7fb4780addb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:47.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.914+0000 7fb47cb21640 1 --2- 172.21.15.29:0/2810775000 >> v2:172.21.15.102:3300/0 conn(0x7fb4780b70f0 0x7fb4780b7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:47.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.914+0000 7fb47d322640 1 -- 172.21.15.29:0/2810775000 >> v2:172.21.15.102:3300/0 conn(0x7fb4780b70f0 msgr2=0x7fb4780b7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:43:47.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.914+0000 7fb47d322640 1 --2- 172.21.15.29:0/2810775000 >> v2:172.21.15.102:3300/0 conn(0x7fb4780b70f0 0x7fb4780b7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:47.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.914+0000 7fb47d322640 1 -- 172.21.15.29:0/2810775000 >> v2:172.21.15.29:3300/0 conn(0x7fb478136f40 msgr2=0x7fb478137350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:43:47.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.914+0000 7fb47d322640 1 --2- 172.21.15.29:0/2810775000 >> v2:172.21.15.29:3300/0 conn(0x7fb478136f40 0x7fb478137350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:47.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.914+0000 7fb47d322640 1 -- 172.21.15.29:0/2810775000 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb478137890 con 0x7fb4780ba100 2024-04-25T19:43:47.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.914+0000 7fb47db23640 1 --2- 172.21.15.29:0/2810775000 >> v2:172.21.15.29:3300/0 conn(0x7fb478136f40 0x7fb478137350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:43:47.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.914+0000 7fb47cb21640 1 --2- 172.21.15.29:0/2810775000 >> v2:172.21.15.102:3300/0 conn(0x7fb4780b70f0 0x7fb4780b7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:43:47.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.915+0000 7fb47d322640 1 --2- 172.21.15.29:0/2810775000 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb4780ba100 0x7fb4780addb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb4780bb050 tx=0x7fb47002bbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d744b5bde2e03039 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:43:47.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.915+0000 7fb477fff640 1 -- 172.21.15.29:0/2810775000 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb470039070 con 0x7fb4780ba100 2024-04-25T19:43:47.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.915+0000 7fb477fff640 1 -- 172.21.15.29:0/2810775000 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb47002c970 con 0x7fb4780ba100 2024-04-25T19:43:47.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.915+0000 7fb477fff640 1 -- 172.21.15.29:0/2810775000 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47000f040 con 0x7fb4780ba100 2024-04-25T19:43:47.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.916+0000 7fb47f586640 1 -- 172.21.15.29:0/2810775000 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb4780ba100 msgr2=0x7fb4780addb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:47.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.916+0000 7fb47f586640 1 --2- 172.21.15.29:0/2810775000 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb4780ba100 0x7fb4780addb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb4780bb050 tx=0x7fb47002bbc0 comp rx=0 tx=0).stop 2024-04-25T19:43:47.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.916+0000 7fb47f586640 1 -- 172.21.15.29:0/2810775000 shutdown_connections 2024-04-25T19:43:47.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.916+0000 7fb47f586640 1 --2- 172.21.15.29:0/2810775000 >> v2:172.21.15.102:3300/0 conn(0x7fb4780b70f0 0x7fb4780b7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:47.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.916+0000 7fb47f586640 1 --2- 172.21.15.29:0/2810775000 >> v2:172.21.15.29:3300/0 conn(0x7fb478136f40 0x7fb478137350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:47.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.916+0000 7fb47f586640 1 --2- 172.21.15.29:0/2810775000 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb4780ba100 0x7fb4780addb0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:47.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.916+0000 7fb47f586640 1 -- 172.21.15.29:0/2810775000 >> 172.21.15.29:0/2810775000 conn(0x7fb478072b90 msgr2=0x7fb4780a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:43:47.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.917+0000 7fb47f586640 1 -- 172.21.15.29:0/2810775000 shutdown_connections 2024-04-25T19:43:47.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.917+0000 7fb47f586640 1 -- 172.21.15.29:0/2810775000 wait complete. 2024-04-25T19:43:47.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.918+0000 7fb47f586640 1 Processor -- start 2024-04-25T19:43:47.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.918+0000 7fb47f586640 1 -- start start 2024-04-25T19:43:47.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.919+0000 7fb47f586640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4780b70f0 0x7fb4780aed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:47.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.919+0000 7fb47f586640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb4780ba100 0x7fb47812e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:47.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.919+0000 7fb47f586640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb478136f40 0x7fb47812e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:47.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.919+0000 7fb47f586640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb4780736c0 con 0x7fb4780b70f0 2024-04-25T19:43:47.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.919+0000 7fb47f586640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb478073540 con 0x7fb478136f40 2024-04-25T19:43:47.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.919+0000 7fb47f586640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fb478073840 con 0x7fb4780ba100 2024-04-25T19:43:47.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.919+0000 7fb47cb21640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb4780ba100 0x7fb47812e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:47.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.919+0000 7fb47d322640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4780b70f0 0x7fb4780aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:47.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.919+0000 7fb47cb21640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb4780ba100 0x7fb47812e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:43300/0 (socket says 172.21.15.29:43300) 2024-04-25T19:43:47.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.919+0000 7fb47d322640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4780b70f0 0x7fb4780aed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58200/0 (socket says 172.21.15.29:58200) 2024-04-25T19:43:47.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.919+0000 7fb47cb21640 1 -- 172.21.15.29:0/437028523 learned_addr learned my addr 172.21.15.29:0/437028523 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:43:47.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.919+0000 7fb47db23640 1 --2- 172.21.15.29:0/437028523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb478136f40 0x7fb47812e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:47.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.920+0000 7fb47d322640 1 -- 172.21.15.29:0/437028523 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb4780ba100 msgr2=0x7fb47812e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:47.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.920+0000 7fb47d322640 1 --2- 172.21.15.29:0/437028523 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb4780ba100 0x7fb47812e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:47.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.920+0000 7fb47d322640 1 -- 172.21.15.29:0/437028523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb478136f40 msgr2=0x7fb47812e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:47.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.920+0000 7fb47d322640 1 --2- 172.21.15.29:0/437028523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb478136f40 0x7fb47812e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.920+0000 7fb47d322640 1 -- 172.21.15.29:0/437028523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4780a9860 con 0x7fb4780b70f0 2024-04-25T19:43:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.920+0000 7fb47d322640 1 --2- 172.21.15.29:0/437028523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4780b70f0 0x7fb4780aed20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb4780ba6c0 tx=0x7fb470004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:43:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.920+0000 7fb475ffb640 1 -- 172.21.15.29:0/437028523 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb470042020 con 0x7fb4780b70f0 2024-04-25T19:43:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.920+0000 7fb475ffb640 1 -- 172.21.15.29:0/437028523 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb47003a040 con 0x7fb4780b70f0 2024-04-25T19:43:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.920+0000 7fb475ffb640 1 -- 172.21.15.29:0/437028523 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb470039070 con 0x7fb4780b70f0 2024-04-25T19:43:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.920+0000 7fb47f586640 1 -- 172.21.15.29:0/437028523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4780af260 con 0x7fb4780b70f0 2024-04-25T19:43:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.920+0000 7fb47f586640 1 -- 172.21.15.29:0/437028523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4781de2b0 con 0x7fb4780b70f0 2024-04-25T19:43:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.920+0000 7fb475ffb640 1 -- 172.21.15.29:0/437028523 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb470045020 con 0x7fb4780b70f0 2024-04-25T19:43:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.920+0000 7fb475ffb640 20 mgrc handle_mgr_map mgrmap(e 1) 2024-04-25T19:43:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.920+0000 7fb475ffb640 4 mgrc handle_mgr_map Got map version 1 2024-04-25T19:43:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.920+0000 7fb475ffb640 4 mgrc handle_mgr_map Active mgr is now 2024-04-25T19:43:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.920+0000 7fb475ffb640 4 mgrc reconnect No active mgr available yet 2024-04-25T19:43:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.920+0000 7fb475ffb640 1 -- 172.21.15.29:0/437028523 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb470047a70 con 0x7fb4780b70f0 2024-04-25T19:43:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.921+0000 7fb47f586640 1 -- 172.21.15.29:0/437028523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4780507b0 con 0x7fb4780b70f0 2024-04-25T19:43:47.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.926+0000 7fb475ffb640 1 -- 172.21.15.29:0/437028523 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fb47004c030 con 0x7fb4780b70f0 2024-04-25T19:43:47.985 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:47.984+0000 7ffac5ec7040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T19:43:47.989 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:47.988+0000 7ff865a32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T19:43:47.990 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:47.988+0000 7fc87723c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T19:43:47.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.992+0000 7fb47f586640 1 -- 172.21.15.29:0/437028523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fb47806f8b0 con 0x7fb4780b70f0 2024-04-25T19:43:47.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.994+0000 7fb475ffb640 1 -- 172.21.15.29:0/437028523 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fb47000f000 con 0x7fb4780b70f0 2024-04-25T19:43:47.995 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2024-04-25T19:43:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.996+0000 7fb47f586640 10 mgrc shutdown 2024-04-25T19:43:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.996+0000 7fb47f586640 1 -- 172.21.15.29:0/437028523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4780b70f0 msgr2=0x7fb4780aed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.996+0000 7fb47f586640 1 --2- 172.21.15.29:0/437028523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4780b70f0 0x7fb4780aed20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb4780ba6c0 tx=0x7fb470004730 comp rx=0 tx=0).stop 2024-04-25T19:43:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.996+0000 7fb47f586640 1 -- 172.21.15.29:0/437028523 shutdown_connections 2024-04-25T19:43:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.996+0000 7fb47f586640 1 --2- 172.21.15.29:0/437028523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb478136f40 0x7fb47812e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.996+0000 7fb47f586640 1 --2- 172.21.15.29:0/437028523 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb4780ba100 0x7fb47812e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.996+0000 7fb47f586640 1 --2- 172.21.15.29:0/437028523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4780b70f0 0x7fb4780aed20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.996+0000 7fb47f586640 1 -- 172.21.15.29:0/437028523 >> 172.21.15.29:0/437028523 conn(0x7fb478072b90 msgr2=0x7fb4780b7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:43:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.996+0000 7fb47f586640 1 -- 172.21.15.29:0/437028523 shutdown_connections 2024-04-25T19:43:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:47.996+0000 7fb47f586640 1 -- 172.21.15.29:0/437028523 wait complete. 2024-04-25T19:43:48.008 INFO:tasks.ceph:check_enable_crimson: False 2024-04-25T19:43:48.008 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-25T19:43:48.008 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-25T19:43:48.009 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-25T19:43:48.032 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T19:43:48.032 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-25T19:43:48.057 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T19:43:48.057 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-25T19:43:48.122 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T19:43:48.122 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-25T19:43:48.143 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:48.143+0000 7ff865a32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:43:48.145 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:48.144+0000 7fc87723c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:43:48.166 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph osd new 4d805223-ee84-494f-bd31-59091906aeb9 0 2024-04-25T19:43:48.186 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:48.185+0000 7ffac5ec7040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:43:48.200 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:48.199+0000 7ff865a32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T19:43:48.202 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:48.200+0000 7fc87723c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T19:43:48.242 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:48.241+0000 7ffac5ec7040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T19:43:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.307+0000 7fd74e044640 1 Processor -- start 2024-04-25T19:43:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.307+0000 7fd74e044640 1 -- start start 2024-04-25T19:43:48.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.308+0000 7fd74e044640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd748072570 0x7fd748072980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:48.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.308+0000 7fd74e044640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd748072ec0 0x7fd7481514d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:48.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.308+0000 7fd74e044640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd748073470 con 0x7fd7481466a0 2024-04-25T19:43:48.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.308+0000 7fd74e044640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd7480aaa10 con 0x7fd748072ec0 2024-04-25T19:43:48.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.308+0000 7fd74e044640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fd748073d90 con 0x7fd748072570 2024-04-25T19:43:48.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.308+0000 7fd747fff640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd748072ec0 0x7fd7481514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:48.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.308+0000 7fd747fff640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd748072ec0 0x7fd7481514d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48060/0 (socket says 172.21.15.102:48060) 2024-04-25T19:43:48.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.308+0000 7fd747fff640 1 -- 172.21.15.102:0/1106073725 learned_addr learned my addr 172.21.15.102:0/1106073725 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T19:43:48.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.308+0000 7fd746ffd640 1 --2- 172.21.15.102:0/1106073725 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd748072570 0x7fd748072980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:48.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.309+0000 7fd7467fc640 1 -- 172.21.15.102:0/1106073725 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2569095709 0 0) 0x7fd748073470 con 0x7fd7481466a0 2024-04-25T19:43:48.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.309+0000 7fd7467fc640 1 -- 172.21.15.102:0/1106073725 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd730003520 con 0x7fd7481466a0 2024-04-25T19:43:48.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.309+0000 7fd747fff640 1 -- 172.21.15.102:0/1106073725 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd748072570 msgr2=0x7fd748072980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:43:48.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.309+0000 7fd747fff640 1 --2- 172.21.15.102:0/1106073725 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd748072570 0x7fd748072980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.309+0000 7fd747fff640 1 -- 172.21.15.102:0/1106073725 >> v1:172.21.15.29:6789/0 conn(0x7fd7481466a0 legacy=0x7fd74806c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:48.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.309+0000 7fd747fff640 1 -- 172.21.15.102:0/1106073725 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd748146ab0 con 0x7fd748072ec0 2024-04-25T19:43:48.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.309+0000 7fd747fff640 1 --2- 172.21.15.102:0/1106073725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd748072ec0 0x7fd7481514d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd740009870 tx=0x7fd74002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72be266878a051a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:43:48.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.310+0000 7fd7467fc640 1 -- 172.21.15.102:0/1106073725 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd740034070 con 0x7fd748072ec0 2024-04-25T19:43:48.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.310+0000 7fd7467fc640 1 -- 172.21.15.102:0/1106073725 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd740030840 con 0x7fd748072ec0 2024-04-25T19:43:48.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.310+0000 7fd7467fc640 1 -- 172.21.15.102:0/1106073725 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd740030c10 con 0x7fd748072ec0 2024-04-25T19:43:48.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.310+0000 7fd74e044640 1 -- 172.21.15.102:0/1106073725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd748072ec0 msgr2=0x7fd7481514d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:48.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.310+0000 7fd74e044640 1 --2- 172.21.15.102:0/1106073725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd748072ec0 0x7fd7481514d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd740009870 tx=0x7fd74002b980 comp rx=0 tx=0).stop 2024-04-25T19:43:48.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.311+0000 7fd74e044640 1 -- 172.21.15.102:0/1106073725 shutdown_connections 2024-04-25T19:43:48.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.311+0000 7fd74e044640 1 --2- 172.21.15.102:0/1106073725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd748072ec0 0x7fd7481514d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.311+0000 7fd74e044640 1 --2- 172.21.15.102:0/1106073725 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd748072570 0x7fd748072980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.311+0000 7fd74e044640 1 -- 172.21.15.102:0/1106073725 >> 172.21.15.102:0/1106073725 conn(0x7fd74806d6a0 msgr2=0x7fd74806dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:43:48.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.311+0000 7fd74e044640 1 -- 172.21.15.102:0/1106073725 shutdown_connections 2024-04-25T19:43:48.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.311+0000 7fd74e044640 1 -- 172.21.15.102:0/1106073725 wait complete. 2024-04-25T19:43:48.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.312+0000 7fd74e044640 1 Processor -- start 2024-04-25T19:43:48.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.312+0000 7fd74e044640 1 -- start start 2024-04-25T19:43:48.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.313+0000 7fd74e044640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd748072570 0x7fd74813bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:48.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.313+0000 7fd74e044640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd748072ec0 0x7fd748148360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:48.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.313+0000 7fd74e044640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7481466a0 0x7fd748145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:48.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.313+0000 7fd74e044640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd748151da0 con 0x7fd748072570 2024-04-25T19:43:48.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.313+0000 7fd74e044640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd748151c20 con 0x7fd7481466a0 2024-04-25T19:43:48.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.313+0000 7fd74e044640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fd748151f20 con 0x7fd748072ec0 2024-04-25T19:43:48.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.313+0000 7fd747fff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7481466a0 0x7fd748145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:48.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.313+0000 7fd7477fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd748072570 0x7fd74813bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:48.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.313+0000 7fd747fff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7481466a0 0x7fd748145a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48064/0 (socket says 172.21.15.102:48064) 2024-04-25T19:43:48.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.313+0000 7fd746ffd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd748072ec0 0x7fd748148360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:48.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.313+0000 7fd747fff640 1 -- 172.21.15.102:0/1276636635 learned_addr learned my addr 172.21.15.102:0/1276636635 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T19:43:48.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.314+0000 7fd7477fe640 1 -- 172.21.15.102:0/1276636635 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd748072ec0 msgr2=0x7fd748148360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:48.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.314+0000 7fd7477fe640 1 --2- 172.21.15.102:0/1276636635 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd748072ec0 0x7fd748148360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.314+0000 7fd7477fe640 1 -- 172.21.15.102:0/1276636635 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7481466a0 msgr2=0x7fd748145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:48.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.314+0000 7fd7477fe640 1 --2- 172.21.15.102:0/1276636635 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7481466a0 0x7fd748145a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.314+0000 7fd7477fe640 1 -- 172.21.15.102:0/1276636635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd748145f80 con 0x7fd748072570 2024-04-25T19:43:48.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.314+0000 7fd746ffd640 1 --2- 172.21.15.102:0/1276636635 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd748072ec0 0x7fd748148360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:43:48.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.314+0000 7fd747fff640 1 --2- 172.21.15.102:0/1276636635 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7481466a0 0x7fd748145a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:43:48.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.314+0000 7fd7477fe640 1 --2- 172.21.15.102:0/1276636635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd748072570 0x7fd74813bc50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd73800c760 tx=0x7fd73800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:43:48.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.314+0000 7fd72ffff640 1 -- 172.21.15.102:0/1276636635 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd738003b70 con 0x7fd748072570 2024-04-25T19:43:48.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.314+0000 7fd72ffff640 1 -- 172.21.15.102:0/1276636635 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd738003d30 con 0x7fd748072570 2024-04-25T19:43:48.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.314+0000 7fd74e044640 1 -- 172.21.15.102:0/1276636635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd748148fb0 con 0x7fd748072570 2024-04-25T19:43:48.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.314+0000 7fd72ffff640 1 -- 172.21.15.102:0/1276636635 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73800aa90 con 0x7fd748072570 2024-04-25T19:43:48.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.314+0000 7fd74e044640 1 -- 172.21.15.102:0/1276636635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7480b54e0 con 0x7fd748072570 2024-04-25T19:43:48.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.315+0000 7fd72ffff640 1 -- 172.21.15.102:0/1276636635 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd73800ac30 con 0x7fd748072570 2024-04-25T19:43:48.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.315+0000 7fd72ffff640 20 mgrc handle_mgr_map mgrmap(e 1) 2024-04-25T19:43:48.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.315+0000 7fd72ffff640 4 mgrc handle_mgr_map Got map version 1 2024-04-25T19:43:48.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.315+0000 7fd72ffff640 4 mgrc handle_mgr_map Active mgr is now 2024-04-25T19:43:48.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.315+0000 7fd72ffff640 4 mgrc reconnect No active mgr available yet 2024-04-25T19:43:48.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.315+0000 7fd72ffff640 1 -- 172.21.15.102:0/1276636635 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fd7380128a0 con 0x7fd748072570 2024-04-25T19:43:48.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.316+0000 7fd74e044640 1 -- 172.21.15.102:0/1276636635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7481488a0 con 0x7fd748072570 2024-04-25T19:43:48.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.318+0000 7fd72ffff640 1 -- 172.21.15.102:0/1276636635 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fd738012af0 con 0x7fd748072570 2024-04-25T19:43:48.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.391+0000 7fd74e044640 1 -- 172.21.15.102:0/1276636635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4d805223-ee84-494f-bd31-59091906aeb9", "id": 0} v 0) -- 0x7fd748149320 con 0x7fd748072570 2024-04-25T19:43:48.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.393+0000 7fd72ffff640 1 -- 172.21.15.102:0/1276636635 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4d805223-ee84-494f-bd31-59091906aeb9", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fd738012cd0 con 0x7fd748072570 2024-04-25T19:43:48.394 INFO:teuthology.orchestra.run.smithi102.stdout:0 2024-04-25T19:43:48.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.394+0000 7fd74e044640 10 mgrc shutdown 2024-04-25T19:43:48.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.394+0000 7fd74e044640 1 -- 172.21.15.102:0/1276636635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd748072570 msgr2=0x7fd74813bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:48.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.394+0000 7fd74e044640 1 --2- 172.21.15.102:0/1276636635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd748072570 0x7fd74813bc50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd73800c760 tx=0x7fd73800cc20 comp rx=0 tx=0).stop 2024-04-25T19:43:48.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.394+0000 7fd74e044640 1 -- 172.21.15.102:0/1276636635 shutdown_connections 2024-04-25T19:43:48.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.394+0000 7fd74e044640 1 --2- 172.21.15.102:0/1276636635 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7481466a0 0x7fd748145a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.394+0000 7fd74e044640 1 --2- 172.21.15.102:0/1276636635 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd748072ec0 0x7fd748148360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.394+0000 7fd74e044640 1 --2- 172.21.15.102:0/1276636635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd748072570 0x7fd74813bc50 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.394+0000 7fd74e044640 1 -- 172.21.15.102:0/1276636635 >> 172.21.15.102:0/1276636635 conn(0x7fd74806d6a0 msgr2=0x7fd7480b9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:43:48.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.394+0000 7fd74e044640 1 -- 172.21.15.102:0/1276636635 shutdown_connections 2024-04-25T19:43:48.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.394+0000 7fd74e044640 1 -- 172.21.15.102:0/1276636635 wait complete. 2024-04-25T19:43:48.406 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph osd new 607c5038-38af-440f-8371-e71758549920 1 2024-04-25T19:43:48.470 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:48.469+0000 7ff865a32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T19:43:48.470 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:48.470+0000 7fc87723c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T19:43:48.508 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:48.506+0000 7ffac5ec7040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T19:43:48.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.514+0000 7fed24a55640 1 Processor -- start 2024-04-25T19:43:48.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.514+0000 7fed24a55640 1 -- start start 2024-04-25T19:43:48.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.515+0000 7fed24a55640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fed200aae50 0x7fed2006c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:48.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.515+0000 7fed24a55640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fed20072dd0 0x7fed201464c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:48.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.515+0000 7fed24a55640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed200731e0 0x7fed2014a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:48.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.515+0000 7fed24a55640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fed20146e60 con 0x7fed20072dd0 2024-04-25T19:43:48.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.515+0000 7fed24a55640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fed2006cbd0 con 0x7fed200aae50 2024-04-25T19:43:48.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.515+0000 7fed24a55640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fed200744b0 con 0x7fed200731e0 2024-04-25T19:43:48.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.515+0000 7fed1e59c640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fed200aae50 0x7fed2006c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:48.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.515+0000 7fed1e59c640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fed200aae50 0x7fed2006c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48080/0 (socket says 172.21.15.102:48080) 2024-04-25T19:43:48.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.515+0000 7fed1e59c640 1 -- 172.21.15.102:0/508132071 learned_addr learned my addr 172.21.15.102:0/508132071 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T19:43:48.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.515+0000 7fed1dd9b640 1 --2- 172.21.15.102:0/508132071 >> v2:172.21.15.29:3300/0 conn(0x7fed20072dd0 0x7fed201464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:48.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.515+0000 7fed1ed9d640 1 --2- 172.21.15.102:0/508132071 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed200731e0 0x7fed2014a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:48.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.516+0000 7fed1e59c640 1 -- 172.21.15.102:0/508132071 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed200731e0 msgr2=0x7fed2014a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:43:48.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.516+0000 7fed1e59c640 1 --2- 172.21.15.102:0/508132071 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed200731e0 0x7fed2014a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.516+0000 7fed1e59c640 1 -- 172.21.15.102:0/508132071 >> v2:172.21.15.29:3300/0 conn(0x7fed20072dd0 msgr2=0x7fed201464c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:43:48.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.516+0000 7fed1e59c640 1 --2- 172.21.15.102:0/508132071 >> v2:172.21.15.29:3300/0 conn(0x7fed20072dd0 0x7fed201464c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.516+0000 7fed1e59c640 1 -- 172.21.15.102:0/508132071 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed200ab260 con 0x7fed200aae50 2024-04-25T19:43:48.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.516+0000 7fed1e59c640 1 --2- 172.21.15.102:0/508132071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed200aae50 0x7fed2006c690 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fed200aaba0 tx=0x7fed1002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9eec93207fae9774 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:43:48.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.517+0000 7fed1d59a640 1 -- 172.21.15.102:0/508132071 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed10035020 con 0x7fed200aae50 2024-04-25T19:43:48.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.517+0000 7fed1d59a640 1 -- 172.21.15.102:0/508132071 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed10031870 con 0x7fed200aae50 2024-04-25T19:43:48.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.517+0000 7fed1d59a640 1 -- 172.21.15.102:0/508132071 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed10031c40 con 0x7fed200aae50 2024-04-25T19:43:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.517+0000 7fed24a55640 1 -- 172.21.15.102:0/508132071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed200aae50 msgr2=0x7fed2006c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.517+0000 7fed24a55640 1 --2- 172.21.15.102:0/508132071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed200aae50 0x7fed2006c690 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fed200aaba0 tx=0x7fed1002b910 comp rx=0 tx=0).stop 2024-04-25T19:43:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.518+0000 7fed24a55640 1 -- 172.21.15.102:0/508132071 shutdown_connections 2024-04-25T19:43:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.518+0000 7fed24a55640 1 --2- 172.21.15.102:0/508132071 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed200731e0 0x7fed2014a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.518+0000 7fed24a55640 1 --2- 172.21.15.102:0/508132071 >> v2:172.21.15.29:3300/0 conn(0x7fed20072dd0 0x7fed201464c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.518+0000 7fed24a55640 1 --2- 172.21.15.102:0/508132071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed200aae50 0x7fed2006c690 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.518+0000 7fed24a55640 1 -- 172.21.15.102:0/508132071 >> 172.21.15.102:0/508132071 conn(0x7fed2006d6a0 msgr2=0x7fed2006dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:43:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.518+0000 7fed24a55640 1 -- 172.21.15.102:0/508132071 shutdown_connections 2024-04-25T19:43:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.518+0000 7fed24a55640 1 -- 172.21.15.102:0/508132071 wait complete. 2024-04-25T19:43:48.520 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:48.518+0000 7ff865a32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T19:43:48.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.519+0000 7fed24a55640 1 Processor -- start 2024-04-25T19:43:48.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.519+0000 7fed24a55640 1 -- start start 2024-04-25T19:43:48.521 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:48.520+0000 7fc87723c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T19:43:48.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.520+0000 7fed24a55640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed20072dd0 0x7fed20141460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:48.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.520+0000 7fed24a55640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed200731e0 0x7fed201dc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:48.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.520+0000 7fed24a55640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed200aae50 0x7fed201e0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:48.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.520+0000 7fed24a55640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fed2014a6f0 con 0x7fed200aae50 2024-04-25T19:43:48.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.520+0000 7fed24a55640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fed20146c10 con 0x7fed20072dd0 2024-04-25T19:43:48.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.520+0000 7fed24a55640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fed2014a870 con 0x7fed200731e0 2024-04-25T19:43:48.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.520+0000 7fed1e59c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed20072dd0 0x7fed20141460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:48.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.521+0000 7fed1e59c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed20072dd0 0x7fed20141460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48084/0 (socket says 172.21.15.102:48084) 2024-04-25T19:43:48.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.521+0000 7fed1e59c640 1 -- 172.21.15.102:0/1437807485 learned_addr learned my addr 172.21.15.102:0/1437807485 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T19:43:48.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.521+0000 7fed1dd9b640 1 --2- 172.21.15.102:0/1437807485 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed200731e0 0x7fed201dc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:48.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.521+0000 7fed1ed9d640 1 --2- 172.21.15.102:0/1437807485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed200aae50 0x7fed201e0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:48.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.521+0000 7fed1e59c640 1 -- 172.21.15.102:0/1437807485 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed200731e0 msgr2=0x7fed201dc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:48.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.521+0000 7fed1e59c640 1 --2- 172.21.15.102:0/1437807485 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed200731e0 0x7fed201dc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.521+0000 7fed1e59c640 1 -- 172.21.15.102:0/1437807485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed200aae50 msgr2=0x7fed201e0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:48.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.521+0000 7fed1e59c640 1 --2- 172.21.15.102:0/1437807485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed200aae50 0x7fed201e0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.521+0000 7fed1e59c640 1 -- 172.21.15.102:0/1437807485 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed201e16a0 con 0x7fed20072dd0 2024-04-25T19:43:48.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.521+0000 7fed1e59c640 1 --2- 172.21.15.102:0/1437807485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed20072dd0 0x7fed20141460 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fed10000c00 tx=0x7fed100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:43:48.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.521+0000 7fed0effd640 1 -- 172.21.15.102:0/1437807485 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed10035040 con 0x7fed20072dd0 2024-04-25T19:43:48.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.521+0000 7fed0effd640 1 -- 172.21.15.102:0/1437807485 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed10030af0 con 0x7fed20072dd0 2024-04-25T19:43:48.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.521+0000 7fed24a55640 1 -- 172.21.15.102:0/1437807485 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed20144380 con 0x7fed20072dd0 2024-04-25T19:43:48.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.521+0000 7fed0effd640 1 -- 172.21.15.102:0/1437807485 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed10043430 con 0x7fed20072dd0 2024-04-25T19:43:48.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.522+0000 7fed0effd640 1 -- 172.21.15.102:0/1437807485 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fed10038020 con 0x7fed20072dd0 2024-04-25T19:43:48.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.522+0000 7fed0effd640 20 mgrc handle_mgr_map mgrmap(e 1) 2024-04-25T19:43:48.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.522+0000 7fed0effd640 4 mgrc handle_mgr_map Got map version 1 2024-04-25T19:43:48.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.522+0000 7fed0effd640 4 mgrc handle_mgr_map Active mgr is now 2024-04-25T19:43:48.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.522+0000 7fed0effd640 4 mgrc reconnect No active mgr available yet 2024-04-25T19:43:48.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.522+0000 7fed24a55640 1 -- 172.21.15.102:0/1437807485 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed201419a0 con 0x7fed20072dd0 2024-04-25T19:43:48.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.522+0000 7fed0effd640 1 -- 172.21.15.102:0/1437807485 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fed10043b90 con 0x7fed20072dd0 2024-04-25T19:43:48.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.523+0000 7fed24a55640 1 -- 172.21.15.102:0/1437807485 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed200507b0 con 0x7fed20072dd0 2024-04-25T19:43:48.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.525+0000 7fed0effd640 1 -- 172.21.15.102:0/1437807485 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fed10043d90 con 0x7fed20072dd0 2024-04-25T19:43:48.556 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:48.555+0000 7ffac5ec7040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T19:43:48.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.598+0000 7fed24a55640 1 -- 172.21.15.102:0/1437807485 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd new", "uuid": "607c5038-38af-440f-8371-e71758549920", "id": 1} v 0) -- 0x7fed200aac30 con 0x7fed20072dd0 2024-04-25T19:43:48.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.607+0000 7fed0effd640 1 -- 172.21.15.102:0/1437807485 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "607c5038-38af-440f-8371-e71758549920", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fed1003a420 con 0x7fed20072dd0 2024-04-25T19:43:48.607 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-04-25T19:43:48.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.607+0000 7fed24a55640 10 mgrc shutdown 2024-04-25T19:43:48.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.607+0000 7fed24a55640 1 -- 172.21.15.102:0/1437807485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed20072dd0 msgr2=0x7fed20141460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:48.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.607+0000 7fed24a55640 1 --2- 172.21.15.102:0/1437807485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed20072dd0 0x7fed20141460 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fed10000c00 tx=0x7fed100046c0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.608+0000 7fed24a55640 1 -- 172.21.15.102:0/1437807485 shutdown_connections 2024-04-25T19:43:48.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.608+0000 7fed24a55640 1 --2- 172.21.15.102:0/1437807485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed200aae50 0x7fed201e0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.608+0000 7fed24a55640 1 --2- 172.21.15.102:0/1437807485 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed200731e0 0x7fed201dc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.608+0000 7fed24a55640 1 --2- 172.21.15.102:0/1437807485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed20072dd0 0x7fed20141460 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.608+0000 7fed24a55640 1 -- 172.21.15.102:0/1437807485 >> 172.21.15.102:0/1437807485 conn(0x7fed2006d6a0 msgr2=0x7fed200a91b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:43:48.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.608+0000 7fed24a55640 1 -- 172.21.15.102:0/1437807485 shutdown_connections 2024-04-25T19:43:48.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.608+0000 7fed24a55640 1 -- 172.21.15.102:0/1437807485 wait complete. 2024-04-25T19:43:48.619 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph osd new 6859ae76-54bb-4c55-b0dc-5e2735c0d31c 2 2024-04-25T19:43:48.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.725+0000 7feeb48ab640 1 Processor -- start 2024-04-25T19:43:48.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.725+0000 7feeb48ab640 1 -- start start 2024-04-25T19:43:48.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.727+0000 7feeb48ab640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7feeb00aa840 0x7feeb00a9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:48.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.727+0000 7feeb48ab640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feeb0147390 con 0x7feeb00a9960 2024-04-25T19:43:48.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.727+0000 7feeb48ab640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feeb0147650 con 0x7feeb0146d20 2024-04-25T19:43:48.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.727+0000 7feeb48ab640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7feeb0147130 con 0x7feeb00aa840 2024-04-25T19:43:48.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.727+0000 7feeae59c640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7feeb00aa840 0x7feeb00a9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:48.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.727+0000 7feeadd9b640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7feeb0146d20 0x7feeb006ff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:48414/0 (socket says 172.21.15.102:48414) 2024-04-25T19:43:48.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.727+0000 7feeadd9b640 1 -- 172.21.15.102:0/3741123062 learned_addr learned my addr 172.21.15.102:0/3741123062 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-04-25T19:43:48.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.728+0000 7feeae59c640 1 -- 172.21.15.102:0/3741123062 >> v1:172.21.15.102:6789/0 conn(0x7feeb0146d20 legacy=0x7feeb006ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:48.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.728+0000 7feeae59c640 1 -- 172.21.15.102:0/3741123062 >> v1:172.21.15.29:6789/0 conn(0x7feeb00a9960 legacy=0x7feeb014e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:48.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.728+0000 7feead59a640 1 -- 172.21.15.102:0/3741123062 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3200947405 0 0) 0x7feeb0147650 con 0x7feeb0146d20 2024-04-25T19:43:48.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.728+0000 7feeae59c640 1 -- 172.21.15.102:0/3741123062 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb00a9d70 con 0x7feeb00aa840 2024-04-25T19:43:48.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.728+0000 7feeae59c640 1 --2- 172.21.15.102:0/3741123062 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeb00aa840 0x7feeb00a9140 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7feea00097a0 tx=0x7feea002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36b647b717641b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:43:48.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.729+0000 7feead59a640 1 -- 172.21.15.102:0/3741123062 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea0034070 con 0x7feeb00aa840 2024-04-25T19:43:48.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.729+0000 7feead59a640 1 -- 172.21.15.102:0/3741123062 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feea0030610 con 0x7feeb00aa840 2024-04-25T19:43:48.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.729+0000 7feead59a640 1 -- 172.21.15.102:0/3741123062 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea00309e0 con 0x7feeb00aa840 2024-04-25T19:43:48.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.729+0000 7feeb48ab640 1 -- 172.21.15.102:0/3741123062 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeb00aa840 msgr2=0x7feeb00a9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:48.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.729+0000 7feeb48ab640 1 --2- 172.21.15.102:0/3741123062 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeb00aa840 0x7feeb00a9140 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7feea00097a0 tx=0x7feea002b960 comp rx=0 tx=0).stop 2024-04-25T19:43:48.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.730+0000 7feeb48ab640 1 -- 172.21.15.102:0/3741123062 shutdown_connections 2024-04-25T19:43:48.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.730+0000 7feeb48ab640 1 --2- 172.21.15.102:0/3741123062 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeb00aa840 0x7feeb00a9140 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.730+0000 7feeb48ab640 1 -- 172.21.15.102:0/3741123062 >> 172.21.15.102:0/3741123062 conn(0x7feeb00a7d40 msgr2=0x7feeb00a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:43:48.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.730+0000 7feeb48ab640 1 -- 172.21.15.102:0/3741123062 shutdown_connections 2024-04-25T19:43:48.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.730+0000 7feeb48ab640 1 -- 172.21.15.102:0/3741123062 wait complete. 2024-04-25T19:43:48.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.731+0000 7feeb48ab640 1 Processor -- start 2024-04-25T19:43:48.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.731+0000 7feeb48ab640 1 -- start start 2024-04-25T19:43:48.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.732+0000 7feeb48ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feeb00a9960 0x7feeb00a4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:48.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.732+0000 7feeb48ab640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeb00aa840 0x7feeb013c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:48.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.732+0000 7feeb48ab640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feeb0146d20 0x7feeb006c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:48.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.732+0000 7feeb48ab640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feeb014f950 con 0x7feeb00a9960 2024-04-25T19:43:48.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.732+0000 7feeb48ab640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7feeb014f7d0 con 0x7feeb00aa840 2024-04-25T19:43:48.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.732+0000 7feeb48ab640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7feeb014fad0 con 0x7feeb0146d20 2024-04-25T19:43:48.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.732+0000 7feeadd9b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeb00aa840 0x7feeb013c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:48.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.732+0000 7feeadd9b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeb00aa840 0x7feeb013c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48106/0 (socket says 172.21.15.102:48106) 2024-04-25T19:43:48.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.733+0000 7feeadd9b640 1 -- 172.21.15.102:0/3460364921 learned_addr learned my addr 172.21.15.102:0/3460364921 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T19:43:48.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.733+0000 7feeae59c640 1 --2- 172.21.15.102:0/3460364921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feeb00a9960 0x7feeb00a4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:48.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.733+0000 7feeaed9d640 1 --2- 172.21.15.102:0/3460364921 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feeb0146d20 0x7feeb006c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:48.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.733+0000 7feeadd9b640 1 -- 172.21.15.102:0/3460364921 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feeb0146d20 msgr2=0x7feeb006c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:48.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.733+0000 7feeadd9b640 1 --2- 172.21.15.102:0/3460364921 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feeb0146d20 0x7feeb006c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.733+0000 7feeadd9b640 1 -- 172.21.15.102:0/3460364921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feeb00a9960 msgr2=0x7feeb00a4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:48.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.733+0000 7feeadd9b640 1 --2- 172.21.15.102:0/3460364921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feeb00a9960 0x7feeb00a4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.733+0000 7feeadd9b640 1 -- 172.21.15.102:0/3460364921 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb013c930 con 0x7feeb00aa840 2024-04-25T19:43:48.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.733+0000 7feeadd9b640 1 --2- 172.21.15.102:0/3460364921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeb00aa840 0x7feeb013c3f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7feea400c390 tx=0x7feea400c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:43:48.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.733+0000 7fee9effd640 1 -- 172.21.15.102:0/3460364921 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea40074e0 con 0x7feeb00aa840 2024-04-25T19:43:48.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.733+0000 7fee9effd640 1 -- 172.21.15.102:0/3460364921 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feea40047e0 con 0x7feeb00aa840 2024-04-25T19:43:48.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.733+0000 7feeb48ab640 1 -- 172.21.15.102:0/3460364921 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeb013f620 con 0x7feeb00aa840 2024-04-25T19:43:48.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.733+0000 7fee9effd640 1 -- 172.21.15.102:0/3460364921 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea4004bd0 con 0x7feeb00aa840 2024-04-25T19:43:48.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.733+0000 7feeb48ab640 1 -- 172.21.15.102:0/3460364921 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeb013ea80 con 0x7feeb00aa840 2024-04-25T19:43:48.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.733+0000 7fee9effd640 1 -- 172.21.15.102:0/3460364921 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7feea4011070 con 0x7feeb00aa840 2024-04-25T19:43:48.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.733+0000 7fee9effd640 20 mgrc handle_mgr_map mgrmap(e 1) 2024-04-25T19:43:48.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.733+0000 7fee9effd640 4 mgrc handle_mgr_map Got map version 1 2024-04-25T19:43:48.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.733+0000 7fee9effd640 4 mgrc handle_mgr_map Active mgr is now 2024-04-25T19:43:48.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.733+0000 7fee9effd640 4 mgrc reconnect No active mgr available yet 2024-04-25T19:43:48.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.733+0000 7fee9effd640 1 -- 172.21.15.102:0/3460364921 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7feea40137e0 con 0x7feeb00aa840 2024-04-25T19:43:48.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.735+0000 7fee9cff9640 1 -- 172.21.15.102:0/3460364921 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee88005080 con 0x7feeb00aa840 2024-04-25T19:43:48.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.737+0000 7fee9effd640 1 -- 172.21.15.102:0/3460364921 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7feea4004d70 con 0x7feeb00aa840 2024-04-25T19:43:48.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.809+0000 7fee9cff9640 1 -- 172.21.15.102:0/3460364921 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6859ae76-54bb-4c55-b0dc-5e2735c0d31c", "id": 2} v 0) -- 0x7fee88005670 con 0x7feeb00aa840 2024-04-25T19:43:48.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.817+0000 7fee9effd640 1 -- 172.21.15.102:0/3460364921 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6859ae76-54bb-4c55-b0dc-5e2735c0d31c", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7feea401ee40 con 0x7feeb00aa840 2024-04-25T19:43:48.818 INFO:teuthology.orchestra.run.smithi102.stdout:2 2024-04-25T19:43:48.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.818+0000 7fee9cff9640 10 mgrc shutdown 2024-04-25T19:43:48.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.818+0000 7fee9cff9640 1 -- 172.21.15.102:0/3460364921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeb00aa840 msgr2=0x7feeb013c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:48.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.818+0000 7fee9cff9640 1 --2- 172.21.15.102:0/3460364921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeb00aa840 0x7feeb013c3f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7feea400c390 tx=0x7feea400c850 comp rx=0 tx=0).stop 2024-04-25T19:43:48.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.818+0000 7fee9cff9640 1 -- 172.21.15.102:0/3460364921 shutdown_connections 2024-04-25T19:43:48.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.818+0000 7fee9cff9640 1 --2- 172.21.15.102:0/3460364921 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feeb0146d20 0x7feeb006c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.818+0000 7fee9cff9640 1 --2- 172.21.15.102:0/3460364921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeb00aa840 0x7feeb013c3f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.818+0000 7fee9cff9640 1 --2- 172.21.15.102:0/3460364921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feeb00a9960 0x7feeb00a4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.818+0000 7fee9cff9640 1 -- 172.21.15.102:0/3460364921 >> 172.21.15.102:0/3460364921 conn(0x7feeb00a7d40 msgr2=0x7feeb0072240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:43:48.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.818+0000 7fee9cff9640 1 -- 172.21.15.102:0/3460364921 shutdown_connections 2024-04-25T19:43:48.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.818+0000 7fee9cff9640 1 -- 172.21.15.102:0/3460364921 wait complete. 2024-04-25T19:43:48.829 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph osd new a98b7b88-5206-4f22-84a6-78212e0d5f1f 3 2024-04-25T19:43:48.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.938+0000 7f7ab5bdb640 1 Processor -- start 2024-04-25T19:43:48.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.938+0000 7f7ab5bdb640 1 -- start start 2024-04-25T19:43:48.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.939+0000 7f7ab5bdb640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7ab00aa840 0x7f7ab00a9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:48.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.939+0000 7f7ab5bdb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7ab0146d20 0x7f7ab00a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:48.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.939+0000 7f7ab5bdb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ab0147390 con 0x7f7ab006ff60 2024-04-25T19:43:48.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.939+0000 7f7ab5bdb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f7ab0147130 con 0x7f7ab0146d20 2024-04-25T19:43:48.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.939+0000 7f7ab5bdb640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f7ab0147650 con 0x7f7ab00aa840 2024-04-25T19:43:48.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.940+0000 7f7aaeffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7ab0146d20 0x7f7ab00a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:48.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.940+0000 7f7aaffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7ab006ff60 0x7f7ab0143710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.102:43348/0 (socket says 172.21.15.102:43348) 2024-04-25T19:43:48.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.940+0000 7f7aaffff640 1 -- 172.21.15.102:0/3565862401 learned_addr learned my addr 172.21.15.102:0/3565862401 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-04-25T19:43:48.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.940+0000 7f7aaeffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7ab0146d20 0x7f7ab00a9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.102:51320/0 (socket says 172.21.15.102:51320) 2024-04-25T19:43:48.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.940+0000 7f7aaf7fe640 1 --2- 172.21.15.102:0/3565862401 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7ab00aa840 0x7f7ab00a9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:48.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.940+0000 7f7aaf7fe640 1 -- 172.21.15.102:0/3565862401 >> v2:172.21.15.29:3300/0 conn(0x7f7ab0146d20 msgr2=0x7f7ab00a9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:43:48.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.940+0000 7f7aaf7fe640 1 --2- 172.21.15.102:0/3565862401 >> v2:172.21.15.29:3300/0 conn(0x7f7ab0146d20 0x7f7ab00a9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.940+0000 7f7aaf7fe640 1 -- 172.21.15.102:0/3565862401 >> v1:172.21.15.29:6789/0 conn(0x7f7ab006ff60 legacy=0x7f7ab0143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:48.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.940+0000 7f7aaf7fe640 1 -- 172.21.15.102:0/3565862401 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ab00aae90 con 0x7f7ab00aa840 2024-04-25T19:43:48.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.940+0000 7f7aaeffd640 1 --2- 172.21.15.102:0/3565862401 >> v2:172.21.15.29:3300/0 conn(0x7f7ab0146d20 0x7f7ab00a9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:43:48.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.941+0000 7f7aaf7fe640 1 --2- 172.21.15.102:0/3565862401 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7ab00aa840 0x7f7ab00a9140 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7aa00097a0 tx=0x7f7aa002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66ae2c1cceed074e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:43:48.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.941+0000 7f7aae7fc640 1 -- 172.21.15.102:0/3565862401 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0035020 con 0x7f7ab00aa840 2024-04-25T19:43:48.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.941+0000 7f7aae7fc640 1 -- 172.21.15.102:0/3565862401 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7aa0031830 con 0x7f7ab00aa840 2024-04-25T19:43:48.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.941+0000 7f7aae7fc640 1 -- 172.21.15.102:0/3565862401 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0031c00 con 0x7f7ab00aa840 2024-04-25T19:43:48.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.942+0000 7f7ab5bdb640 1 -- 172.21.15.102:0/3565862401 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7ab00aa840 msgr2=0x7f7ab00a9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:48.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.942+0000 7f7ab5bdb640 1 --2- 172.21.15.102:0/3565862401 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7ab00aa840 0x7f7ab00a9140 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7aa00097a0 tx=0x7f7aa002b8d0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.942+0000 7f7ab5bdb640 1 -- 172.21.15.102:0/3565862401 shutdown_connections 2024-04-25T19:43:48.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.942+0000 7f7ab5bdb640 1 --2- 172.21.15.102:0/3565862401 >> v2:172.21.15.29:3300/0 conn(0x7f7ab0146d20 0x7f7ab00a9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.942+0000 7f7ab5bdb640 1 --2- 172.21.15.102:0/3565862401 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7ab00aa840 0x7f7ab00a9140 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.942+0000 7f7ab5bdb640 1 -- 172.21.15.102:0/3565862401 >> 172.21.15.102:0/3565862401 conn(0x7f7ab00a7d40 msgr2=0x7f7ab00a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:43:48.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.942+0000 7f7ab5bdb640 1 -- 172.21.15.102:0/3565862401 shutdown_connections 2024-04-25T19:43:48.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.942+0000 7f7ab5bdb640 1 -- 172.21.15.102:0/3565862401 wait complete. 2024-04-25T19:43:48.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.943+0000 7f7ab5bdb640 1 Processor -- start 2024-04-25T19:43:48.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.943+0000 7f7ab5bdb640 1 -- start start 2024-04-25T19:43:48.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.944+0000 7f7ab5bdb640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ab006ff60 0x7f7ab00a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:48.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.944+0000 7f7ab5bdb640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7ab00aa840 0x7f7ab00a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:48.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.944+0000 7f7ab5bdb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ab0146d20 0x7f7ab013c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:48.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.944+0000 7f7ab5bdb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab0143e20 con 0x7f7ab0146d20 2024-04-25T19:43:48.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.944+0000 7f7ab5bdb640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab0070580 con 0x7f7ab006ff60 2024-04-25T19:43:48.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.944+0000 7f7ab5bdb640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f7ab0143fa0 con 0x7f7ab00aa840 2024-04-25T19:43:48.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.944+0000 7f7aaf7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ab006ff60 0x7f7ab00a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:48.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.944+0000 7f7aaf7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ab006ff60 0x7f7ab00a4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48108/0 (socket says 172.21.15.102:48108) 2024-04-25T19:43:48.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.944+0000 7f7aaf7fe640 1 -- 172.21.15.102:0/2360301920 learned_addr learned my addr 172.21.15.102:0/2360301920 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T19:43:48.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.944+0000 7f7aaeffd640 1 --2- 172.21.15.102:0/2360301920 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7ab00aa840 0x7f7ab00a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:48.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.944+0000 7f7aaffff640 1 --2- 172.21.15.102:0/2360301920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ab0146d20 0x7f7ab013c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:48.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.945+0000 7f7aaf7fe640 1 -- 172.21.15.102:0/2360301920 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7ab00aa840 msgr2=0x7f7ab00a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:48.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.945+0000 7f7aaf7fe640 1 --2- 172.21.15.102:0/2360301920 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7ab00aa840 0x7f7ab00a4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.945+0000 7f7aaf7fe640 1 -- 172.21.15.102:0/2360301920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ab0146d20 msgr2=0x7f7ab013c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:48.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.945+0000 7f7aaf7fe640 1 --2- 172.21.15.102:0/2360301920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ab0146d20 0x7f7ab013c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:48.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.945+0000 7f7aaf7fe640 1 -- 172.21.15.102:0/2360301920 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ab013ca80 con 0x7f7ab006ff60 2024-04-25T19:43:48.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.945+0000 7f7aaeffd640 1 --2- 172.21.15.102:0/2360301920 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7ab00aa840 0x7f7ab00a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:43:48.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.945+0000 7f7aaf7fe640 1 --2- 172.21.15.102:0/2360301920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ab006ff60 0x7f7ab00a4690 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0000c00 tx=0x7f7aa00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:43:48.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.945+0000 7f7a97fff640 1 -- 172.21.15.102:0/2360301920 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0040070 con 0x7f7ab006ff60 2024-04-25T19:43:48.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.945+0000 7f7a97fff640 1 -- 172.21.15.102:0/2360301920 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7aa003a930 con 0x7f7ab006ff60 2024-04-25T19:43:48.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.945+0000 7f7a97fff640 1 -- 172.21.15.102:0/2360301920 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0035040 con 0x7f7ab006ff60 2024-04-25T19:43:48.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.945+0000 7f7ab5bdb640 1 -- 172.21.15.102:0/2360301920 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ab00b11c0 con 0x7f7ab006ff60 2024-04-25T19:43:48.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.945+0000 7f7ab5bdb640 1 -- 172.21.15.102:0/2360301920 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ab00b8b90 con 0x7f7ab006ff60 2024-04-25T19:43:48.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.945+0000 7f7a97fff640 1 -- 172.21.15.102:0/2360301920 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7aa004b060 con 0x7f7ab006ff60 2024-04-25T19:43:48.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.945+0000 7f7a97fff640 20 mgrc handle_mgr_map mgrmap(e 1) 2024-04-25T19:43:48.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.945+0000 7f7a97fff640 4 mgrc handle_mgr_map Got map version 1 2024-04-25T19:43:48.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.945+0000 7f7a97fff640 4 mgrc handle_mgr_map Active mgr is now 2024-04-25T19:43:48.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.945+0000 7f7a97fff640 4 mgrc reconnect No active mgr available yet 2024-04-25T19:43:48.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.945+0000 7f7a97fff640 1 -- 172.21.15.102:0/2360301920 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f7aa003adc0 con 0x7f7ab006ff60 2024-04-25T19:43:48.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.946+0000 7f7ab5bdb640 1 -- 172.21.15.102:0/2360301920 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ab00507b0 con 0x7f7ab006ff60 2024-04-25T19:43:48.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:48.948+0000 7f7a97fff640 1 -- 172.21.15.102:0/2360301920 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f7aa0044400 con 0x7f7ab006ff60 2024-04-25T19:43:49.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:49.020+0000 7f7ab5bdb640 1 -- 172.21.15.102:0/2360301920 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a98b7b88-5206-4f22-84a6-78212e0d5f1f", "id": 3} v 0) -- 0x7f7ab00a6d00 con 0x7f7ab006ff60 2024-04-25T19:43:49.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:49.029+0000 7f7a97fff640 1 -- 172.21.15.102:0/2360301920 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a98b7b88-5206-4f22-84a6-78212e0d5f1f", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f7aa004d7e0 con 0x7f7ab006ff60 2024-04-25T19:43:49.030 INFO:teuthology.orchestra.run.smithi102.stdout:3 2024-04-25T19:43:49.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:49.029+0000 7f7ab5bdb640 10 mgrc shutdown 2024-04-25T19:43:49.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:49.030+0000 7f7ab5bdb640 1 -- 172.21.15.102:0/2360301920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ab006ff60 msgr2=0x7f7ab00a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:49.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:49.030+0000 7f7ab5bdb640 1 --2- 172.21.15.102:0/2360301920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ab006ff60 0x7f7ab00a4690 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0000c00 tx=0x7f7aa00046c0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:49.030+0000 7f7ab5bdb640 1 -- 172.21.15.102:0/2360301920 shutdown_connections 2024-04-25T19:43:49.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:49.030+0000 7f7ab5bdb640 1 --2- 172.21.15.102:0/2360301920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ab0146d20 0x7f7ab013c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:49.030+0000 7f7ab5bdb640 1 --2- 172.21.15.102:0/2360301920 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7ab00aa840 0x7f7ab00a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:49.030+0000 7f7ab5bdb640 1 --2- 172.21.15.102:0/2360301920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ab006ff60 0x7f7ab00a4690 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:49.030+0000 7f7ab5bdb640 1 -- 172.21.15.102:0/2360301920 >> 172.21.15.102:0/2360301920 conn(0x7f7ab00a7d40 msgr2=0x7f7ab0134700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:43:49.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:49.030+0000 7f7ab5bdb640 1 -- 172.21.15.102:0/2360301920 shutdown_connections 2024-04-25T19:43:49.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T19:43:49.030+0000 7f7ab5bdb640 1 -- 172.21.15.102:0/2360301920 wait complete. 2024-04-25T19:43:49.043 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-25T19:43:49.043 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-25T19:43:49.045 INFO:tasks.ceph.osd.0:Started 2024-04-25T19:43:49.045 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-25T19:43:49.046 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-25T19:43:49.048 INFO:tasks.ceph.osd.1:Started 2024-04-25T19:43:49.048 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-25T19:43:49.048 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-25T19:43:49.050 INFO:tasks.ceph.osd.2:Started 2024-04-25T19:43:49.050 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-25T19:43:49.050 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-25T19:43:49.052 INFO:tasks.ceph.osd.3:Started 2024-04-25T19:43:49.053 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:43:49.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.169+0000 7f4b24d79640 1 Processor -- start 2024-04-25T19:43:49.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.169+0000 7f4b24d79640 1 -- start start 2024-04-25T19:43:49.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.170+0000 7f4b24d79640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4b201466a0 0x7f4b2006c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:49.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.170+0000 7f4b24d79640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4b20072570 0x7f4b20072980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:49.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.170+0000 7f4b24d79640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4b20072ec0 0x7f4b200a92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:49.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.170+0000 7f4b24d79640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f4b20073470 con 0x7f4b201466a0 2024-04-25T19:43:49.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.170+0000 7f4b24d79640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f4b200aaa10 con 0x7f4b20072ec0 2024-04-25T19:43:49.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.170+0000 7f4b24d79640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f4b2006cad0 con 0x7f4b20072570 2024-04-25T19:43:49.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.170+0000 7f4b1e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4b201466a0 0x7f4b2006c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:49.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.170+0000 7f4b1dd9b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4b20072570 0x7f4b20072980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:49.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.170+0000 7f4b1e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4b201466a0 0x7f4b2006c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58208/0 (socket says 172.21.15.29:58208) 2024-04-25T19:43:49.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.170+0000 7f4b1dd9b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4b20072570 0x7f4b20072980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:43314/0 (socket says 172.21.15.29:43314) 2024-04-25T19:43:49.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.170+0000 7f4b1e59c640 1 -- 172.21.15.29:0/1291983756 learned_addr learned my addr 172.21.15.29:0/1291983756 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:43:49.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.170+0000 7f4b1ed9d640 1 --2- 172.21.15.29:0/1291983756 >> v2:172.21.15.102:3300/0 conn(0x7f4b20072ec0 0x7f4b200a92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:49.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.170+0000 7f4b1dd9b640 1 -- 172.21.15.29:0/1291983756 >> v2:172.21.15.102:3300/0 conn(0x7f4b20072ec0 msgr2=0x7f4b200a92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:43:49.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.170+0000 7f4b1dd9b640 1 --2- 172.21.15.29:0/1291983756 >> v2:172.21.15.102:3300/0 conn(0x7f4b20072ec0 0x7f4b200a92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.170+0000 7f4b1dd9b640 1 -- 172.21.15.29:0/1291983756 >> v2:172.21.15.29:3300/0 conn(0x7f4b201466a0 msgr2=0x7f4b2006c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:43:49.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.170+0000 7f4b1dd9b640 1 --2- 172.21.15.29:0/1291983756 >> v2:172.21.15.29:3300/0 conn(0x7f4b201466a0 0x7f4b2006c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.170+0000 7f4b1dd9b640 1 -- 172.21.15.29:0/1291983756 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b20073d90 con 0x7f4b20072570 2024-04-25T19:43:49.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.170+0000 7f4b1e59c640 1 --2- 172.21.15.29:0/1291983756 >> v2:172.21.15.29:3300/0 conn(0x7f4b201466a0 0x7f4b2006c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:43:49.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.171+0000 7f4b1dd9b640 1 --2- 172.21.15.29:0/1291983756 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4b20072570 0x7f4b20072980 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4b200ab430 tx=0x7f4b1002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e93413520ef44d0c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:43:49.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.171+0000 7f4b1d59a640 1 -- 172.21.15.29:0/1291983756 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b100315a0 con 0x7f4b20072570 2024-04-25T19:43:49.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.171+0000 7f4b1d59a640 1 -- 172.21.15.29:0/1291983756 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b10037070 con 0x7f4b20072570 2024-04-25T19:43:49.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.171+0000 7f4b1d59a640 1 -- 172.21.15.29:0/1291983756 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b10031960 con 0x7f4b20072570 2024-04-25T19:43:49.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.172+0000 7f4b24d79640 1 -- 172.21.15.29:0/1291983756 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4b20072570 msgr2=0x7f4b20072980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:49.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.172+0000 7f4b24d79640 1 --2- 172.21.15.29:0/1291983756 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4b20072570 0x7f4b20072980 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4b200ab430 tx=0x7f4b1002b840 comp rx=0 tx=0).stop 2024-04-25T19:43:49.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.172+0000 7f4b24d79640 1 -- 172.21.15.29:0/1291983756 shutdown_connections 2024-04-25T19:43:49.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.172+0000 7f4b24d79640 1 --2- 172.21.15.29:0/1291983756 >> v2:172.21.15.102:3300/0 conn(0x7f4b20072ec0 0x7f4b200a92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.172+0000 7f4b24d79640 1 --2- 172.21.15.29:0/1291983756 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4b20072570 0x7f4b20072980 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.172+0000 7f4b24d79640 1 --2- 172.21.15.29:0/1291983756 >> v2:172.21.15.29:3300/0 conn(0x7f4b201466a0 0x7f4b2006c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.172+0000 7f4b24d79640 1 -- 172.21.15.29:0/1291983756 >> 172.21.15.29:0/1291983756 conn(0x7f4b2006d6a0 msgr2=0x7f4b2006dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:43:49.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.172+0000 7f4b24d79640 1 -- 172.21.15.29:0/1291983756 shutdown_connections 2024-04-25T19:43:49.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.172+0000 7f4b24d79640 1 -- 172.21.15.29:0/1291983756 wait complete. 2024-04-25T19:43:49.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.174+0000 7f4b24d79640 1 Processor -- start 2024-04-25T19:43:49.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.174+0000 7f4b24d79640 1 -- start start 2024-04-25T19:43:49.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.175+0000 7f4b24d79640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b20072570 0x7f4b2009b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:49.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.175+0000 7f4b24d79640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4b20072ec0 0x7f4b20140ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:49.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.175+0000 7f4b24d79640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b201466a0 0x7f4b2009aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:49.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.175+0000 7f4b24d79640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4b200a9970 con 0x7f4b201466a0 2024-04-25T19:43:49.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.175+0000 7f4b24d79640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4b200a97f0 con 0x7f4b20072570 2024-04-25T19:43:49.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.175+0000 7f4b24d79640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f4b200a9af0 con 0x7f4b20072ec0 2024-04-25T19:43:49.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.175+0000 7f4b1dd9b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4b20072ec0 0x7f4b20140ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:49.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.175+0000 7f4b1ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b201466a0 0x7f4b2009aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:49.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.175+0000 7f4b1dd9b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4b20072ec0 0x7f4b20140ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:43324/0 (socket says 172.21.15.29:43324) 2024-04-25T19:43:49.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.175+0000 7f4b1dd9b640 1 -- 172.21.15.29:0/4104851977 learned_addr learned my addr 172.21.15.29:0/4104851977 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:43:49.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.175+0000 7f4b1e59c640 1 --2- 172.21.15.29:0/4104851977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b20072570 0x7f4b2009b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:49.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.175+0000 7f4b1dd9b640 1 -- 172.21.15.29:0/4104851977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b20072570 msgr2=0x7f4b2009b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:49.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.175+0000 7f4b1dd9b640 1 --2- 172.21.15.29:0/4104851977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b20072570 0x7f4b2009b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.175+0000 7f4b1dd9b640 1 -- 172.21.15.29:0/4104851977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b201466a0 msgr2=0x7f4b2009aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:49.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.175+0000 7f4b1dd9b640 1 --2- 172.21.15.29:0/4104851977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b201466a0 0x7f4b2009aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.175+0000 7f4b1dd9b640 1 -- 172.21.15.29:0/4104851977 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b201cef80 con 0x7f4b20072ec0 2024-04-25T19:43:49.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.176+0000 7f4b1dd9b640 1 --2- 172.21.15.29:0/4104851977 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4b20072ec0 0x7f4b20140ce0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4b10000c00 tx=0x7f4b100046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:43:49.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.176+0000 7f4b0effd640 1 -- 172.21.15.29:0/4104851977 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b10032630 con 0x7f4b20072ec0 2024-04-25T19:43:49.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.176+0000 7f4b0effd640 1 -- 172.21.15.29:0/4104851977 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b100327f0 con 0x7f4b20072ec0 2024-04-25T19:43:49.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.176+0000 7f4b0effd640 1 -- 172.21.15.29:0/4104851977 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b10032be0 con 0x7f4b20072ec0 2024-04-25T19:43:49.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.176+0000 7f4b24d79640 1 -- 172.21.15.29:0/4104851977 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b20141220 con 0x7f4b20072ec0 2024-04-25T19:43:49.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.176+0000 7f4b24d79640 1 -- 172.21.15.29:0/4104851977 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b2009c130 con 0x7f4b20072ec0 2024-04-25T19:43:49.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.176+0000 7f4b0effd640 1 -- 172.21.15.29:0/4104851977 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4b10045780 con 0x7f4b20072ec0 2024-04-25T19:43:49.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.176+0000 7f4b0effd640 20 mgrc handle_mgr_map mgrmap(e 1) 2024-04-25T19:43:49.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.176+0000 7f4b0effd640 4 mgrc handle_mgr_map Got map version 1 2024-04-25T19:43:49.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.176+0000 7f4b0effd640 4 mgrc handle_mgr_map Active mgr is now 2024-04-25T19:43:49.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.176+0000 7f4b0effd640 4 mgrc reconnect No active mgr available yet 2024-04-25T19:43:49.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.176+0000 7f4b0effd640 1 -- 172.21.15.29:0/4104851977 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f4b1002f030 con 0x7f4b20072ec0 2024-04-25T19:43:49.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.177+0000 7f4b24d79640 1 -- 172.21.15.29:0/4104851977 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b200507b0 con 0x7f4b20072ec0 2024-04-25T19:43:49.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.179+0000 7f4b0effd640 1 -- 172.21.15.29:0/4104851977 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f4b10040070 con 0x7f4b20072ec0 2024-04-25T19:43:49.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.241+0000 7f4b24d79640 1 -- 172.21.15.29:0/4104851977 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4b200aac70 con 0x7f4b20072ec0 2024-04-25T19:43:49.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.242+0000 7f4b0effd640 1 -- 172.21.15.29:0/4104851977 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v6) ==== 74+0+4153 (secure 0 0 0) 0x7f4b10052090 con 0x7f4b20072ec0 2024-04-25T19:43:49.243 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:43:49.243 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":6,"fsid":"1b3c39b5-e787-409e-b0bd-f4f491c89d72","created":"2024-04-25T19:43:47.854697+0000","modified":"2024-04-25T19:43:49.022776+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T19:43:49.022776+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4d805223-ee84-494f-bd31-59091906aeb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"607c5038-38af-440f-8371-e71758549920","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6859ae76-54bb-4c55-b0dc-5e2735c0d31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a98b7b88-5206-4f22-84a6-78212e0d5f1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:43:49.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.243+0000 7f4b24d79640 10 mgrc shutdown 2024-04-25T19:43:49.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.243+0000 7f4b24d79640 1 -- 172.21.15.29:0/4104851977 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4b20072ec0 msgr2=0x7f4b20140ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:49.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.243+0000 7f4b24d79640 1 --2- 172.21.15.29:0/4104851977 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4b20072ec0 0x7f4b20140ce0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4b10000c00 tx=0x7f4b100046c0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.243+0000 7f4b24d79640 1 -- 172.21.15.29:0/4104851977 shutdown_connections 2024-04-25T19:43:49.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.243+0000 7f4b24d79640 1 --2- 172.21.15.29:0/4104851977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b201466a0 0x7f4b2009aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.243+0000 7f4b24d79640 1 --2- 172.21.15.29:0/4104851977 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4b20072ec0 0x7f4b20140ce0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.243+0000 7f4b24d79640 1 --2- 172.21.15.29:0/4104851977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b20072570 0x7f4b2009b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.243+0000 7f4b24d79640 1 -- 172.21.15.29:0/4104851977 >> 172.21.15.29:0/4104851977 conn(0x7f4b2006d6a0 msgr2=0x7f4b200b9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:43:49.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.243+0000 7f4b24d79640 1 -- 172.21.15.29:0/4104851977 shutdown_connections 2024-04-25T19:43:49.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:43:49.243+0000 7f4b24d79640 1 -- 172.21.15.29:0/4104851977 wait complete. 2024-04-25T19:43:49.255 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-25T19:43:49.256 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-25T19:43:49.301 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:49.301+0000 7ff865a32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T19:43:49.302 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:49.301+0000 7fc87723c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T19:43:49.352 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:49.351+0000 7ffac5ec7040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T19:43:49.390 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:49.390+0000 7ff865a32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T19:43:49.391 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:49.390+0000 7fc87723c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T19:43:49.435 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:49.434+0000 7ff865a32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T19:43:49.435 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:49.434+0000 7fc87723c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T19:43:49.444 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:49.443+0000 7ffac5ec7040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T19:43:49.489 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:49.488+0000 7ffac5ec7040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T19:43:49.602 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:49.601+0000 7ff865a32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T19:43:49.603 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:49.601+0000 7fc87723c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T19:43:49.649 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-04-25T19:43:49.648+0000 7f4ab0843640 -1 Falling back to public interface 2024-04-25T19:43:49.654 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:49.654+0000 7ff865a32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T19:43:49.655 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:49.654+0000 7fc87723c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T19:43:49.658 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T19:43:49.661 INFO:tasks.ceph.osd.2.smithi102.stderr:2024-04-25T19:43:49.660+0000 7f413c01e640 -1 Falling back to public interface 2024-04-25T19:43:49.662 INFO:tasks.ceph.osd.3.smithi102.stderr:2024-04-25T19:43:49.661+0000 7f054dc12640 -1 Falling back to public interface 2024-04-25T19:43:49.662 INFO:tasks.ceph.osd.3.smithi102.stderr:2024-04-25T19:43:49.661+0000 7f054c56d640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6804/0: (98) Address already in use 2024-04-25T19:43:49.662 INFO:tasks.ceph.osd.1.smithi102.stderr:2024-04-25T19:43:49.661+0000 7f635c101640 -1 Falling back to public interface 2024-04-25T19:43:49.662 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:49.662+0000 7ffac5ec7040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T19:43:49.710 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:49.709+0000 7ff865a32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T19:43:49.711 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:49.710+0000 7fc87723c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T19:43:49.713 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:49.713+0000 7ffac5ec7040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T19:43:49.760 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.759+0000 7f641bfb6640 1 Processor -- start 2024-04-25T19:43:49.760 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.759+0000 7f641bfb6640 1 -- start start 2024-04-25T19:43:49.761 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.760+0000 7f641bfb6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f64140a92b0 0x7f64140acbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:49.761 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.760+0000 7f641bfb6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64140aa770 con 0x7f64140ad120 2024-04-25T19:43:49.761 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.760+0000 7f641bfb6640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6414072b30 con 0x7f64140aa1f0 2024-04-25T19:43:49.762 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.760+0000 7f641bfb6640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f64140a96c0 con 0x7f64140a92b0 2024-04-25T19:43:49.762 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.760+0000 7f6419d52640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f64140a92b0 0x7f64140acbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:49.762 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.760+0000 7f6419551640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f64140ad120 0x7f64140a9ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58642/0 (socket says 172.21.15.29:58642) 2024-04-25T19:43:49.762 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.760+0000 7f6419551640 1 -- 172.21.15.29:0/1533281683 learned_addr learned my addr 172.21.15.29:0/1533281683 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:43:49.763 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.761+0000 7f6419d52640 1 -- 172.21.15.29:0/1533281683 >> v1:172.21.15.102:6789/0 conn(0x7f64140aa1f0 legacy=0x7f641414e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:49.763 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.761+0000 7f6419d52640 1 -- 172.21.15.29:0/1533281683 >> v1:172.21.15.29:6789/0 conn(0x7f64140ad120 legacy=0x7f64140a9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:49.763 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.761+0000 7f6419d52640 1 -- 172.21.15.29:0/1533281683 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64140ad530 con 0x7f64140a92b0 2024-04-25T19:43:49.763 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.761+0000 7f6419d52640 1 --2- 172.21.15.29:0/1533281683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64140a92b0 0x7f64140acbe0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f64040097d0 tx=0x7f640402b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8c0f8d5672996c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:43:49.763 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.762+0000 7f6418d50640 1 -- 172.21.15.29:0/1533281683 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6404034070 con 0x7f64140a92b0 2024-04-25T19:43:49.763 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.762+0000 7f6418d50640 1 -- 172.21.15.29:0/1533281683 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6404030890 con 0x7f64140a92b0 2024-04-25T19:43:49.763 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.762+0000 7f6418d50640 1 -- 172.21.15.29:0/1533281683 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6404030c60 con 0x7f64140a92b0 2024-04-25T19:43:49.764 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.762+0000 7f641bfb6640 1 -- 172.21.15.29:0/1533281683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64140a92b0 msgr2=0x7f64140acbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:49.764 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.762+0000 7f641bfb6640 1 --2- 172.21.15.29:0/1533281683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64140a92b0 0x7f64140acbe0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f64040097d0 tx=0x7f640402b9d0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.764 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.763+0000 7f641bfb6640 1 -- 172.21.15.29:0/1533281683 shutdown_connections 2024-04-25T19:43:49.764 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.763+0000 7f641bfb6640 1 --2- 172.21.15.29:0/1533281683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64140a92b0 0x7f64140acbe0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.764 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.763+0000 7f641bfb6640 1 -- 172.21.15.29:0/1533281683 >> 172.21.15.29:0/1533281683 conn(0x7f641406d6a0 msgr2=0x7f641406dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:43:49.764 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.763+0000 7f641bfb6640 1 -- 172.21.15.29:0/1533281683 shutdown_connections 2024-04-25T19:43:49.764 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.763+0000 7f641bfb6640 1 -- 172.21.15.29:0/1533281683 wait complete. 2024-04-25T19:43:49.765 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.764+0000 7f641bfb6640 1 Processor -- start 2024-04-25T19:43:49.765 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.764+0000 7f641bfb6640 1 -- start start 2024-04-25T19:43:49.768 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.765+0000 7f641bfb6640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f64140a92b0 0x7f64140af300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:49.768 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.765+0000 7f641bfb6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64140aa1f0 0x7f64141dd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:49.768 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.765+0000 7f641bfb6640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64140ad120 0x7f64141dd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:49.768 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.765+0000 7f641bfb6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f641414f9e0 con 0x7f64140aa1f0 2024-04-25T19:43:49.768 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.765+0000 7f641bfb6640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f641414f860 con 0x7f64140ad120 2024-04-25T19:43:49.769 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.765+0000 7f641bfb6640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f641414fb60 con 0x7f64140a92b0 2024-04-25T19:43:49.769 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.765+0000 7f6419551640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64140aa1f0 0x7f64141dd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:49.769 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.765+0000 7f6419d52640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f64140a92b0 0x7f64140af300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:49.770 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.765+0000 7f6419551640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64140aa1f0 0x7f64141dd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58226/0 (socket says 172.21.15.29:58226) 2024-04-25T19:43:49.770 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.765+0000 7f6419551640 1 -- 172.21.15.29:0/212426973 learned_addr learned my addr 172.21.15.29:0/212426973 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:43:49.770 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.765+0000 7f641a553640 1 --2- 172.21.15.29:0/212426973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64140ad120 0x7f64141dd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:49.770 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.766+0000 7f6419d52640 1 -- 172.21.15.29:0/212426973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64140ad120 msgr2=0x7f64141dd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:49.770 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.766+0000 7f6419d52640 1 --2- 172.21.15.29:0/212426973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64140ad120 0x7f64141dd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.770 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.766+0000 7f6419d52640 1 -- 172.21.15.29:0/212426973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64140aa1f0 msgr2=0x7f64141dd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:49.770 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.766+0000 7f6419d52640 1 --2- 172.21.15.29:0/212426973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64140aa1f0 0x7f64141dd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.770 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.766+0000 7f6419d52640 1 -- 172.21.15.29:0/212426973 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6414145220 con 0x7f64140a92b0 2024-04-25T19:43:49.771 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.766+0000 7f6419551640 1 --2- 172.21.15.29:0/212426973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64140aa1f0 0x7f64141dd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:43:49.771 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.766+0000 7f6419d52640 1 --2- 172.21.15.29:0/212426973 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f64140a92b0 0x7f64140af300 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6404033000 tx=0x7f64040046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:43:49.771 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.766+0000 7f640a7fc640 1 -- 172.21.15.29:0/212426973 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6404034040 con 0x7f64140a92b0 2024-04-25T19:43:49.771 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.766+0000 7f640a7fc640 1 -- 172.21.15.29:0/212426973 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64040443f0 con 0x7f64140a92b0 2024-04-25T19:43:49.771 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.766+0000 7f641bfb6640 1 -- 172.21.15.29:0/212426973 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64141de3a0 con 0x7f64140a92b0 2024-04-25T19:43:49.771 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.766+0000 7f640a7fc640 1 -- 172.21.15.29:0/212426973 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64040447e0 con 0x7f64140a92b0 2024-04-25T19:43:49.771 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.766+0000 7f641bfb6640 1 -- 172.21.15.29:0/212426973 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f64140afa20 con 0x7f64140a92b0 2024-04-25T19:43:49.771 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.766+0000 7f640a7fc640 1 -- 172.21.15.29:0/212426973 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f640404b060 con 0x7f64140a92b0 2024-04-25T19:43:49.771 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.766+0000 7f640a7fc640 20 mgrc handle_mgr_map mgrmap(e 1) 2024-04-25T19:43:49.772 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.766+0000 7f640a7fc640 4 mgrc handle_mgr_map Got map version 1 2024-04-25T19:43:49.772 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.766+0000 7f640a7fc640 4 mgrc handle_mgr_map Active mgr is now 2024-04-25T19:43:49.772 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.766+0000 7f640a7fc640 4 mgrc reconnect No active mgr available yet 2024-04-25T19:43:49.772 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.766+0000 7f640a7fc640 1 -- 172.21.15.29:0/212426973 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f640402f8f0 con 0x7f64140a92b0 2024-04-25T19:43:49.772 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.767+0000 7f641bfb6640 1 -- 172.21.15.29:0/212426973 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64140b9870 con 0x7f64140a92b0 2024-04-25T19:43:49.772 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.770+0000 7f640a7fc640 1 -- 172.21.15.29:0/212426973 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f6404062e50 con 0x7f64140a92b0 2024-04-25T19:43:49.773 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:49.772+0000 7ffac5ec7040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T19:43:49.825 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:49.824+0000 7ff865a32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:43:49.826 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:49.825+0000 7fc87723c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:43:49.833 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.832+0000 7f641bfb6640 1 -- 172.21.15.29:0/212426973 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f64140aed00 con 0x7f64140a92b0 2024-04-25T19:43:49.833 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.833+0000 7f640a7fc640 1 -- 172.21.15.29:0/212426973 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v6) ==== 74+0+4153 (secure 0 0 0) 0x7f6404051070 con 0x7f64140a92b0 2024-04-25T19:43:49.834 INFO:teuthology.misc.health.smithi029.stdout: 2024-04-25T19:43:49.834 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":6,"fsid":"1b3c39b5-e787-409e-b0bd-f4f491c89d72","created":"2024-04-25T19:43:47.854697+0000","modified":"2024-04-25T19:43:49.022776+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T19:43:49.022776+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4d805223-ee84-494f-bd31-59091906aeb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"607c5038-38af-440f-8371-e71758549920","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6859ae76-54bb-4c55-b0dc-5e2735c0d31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a98b7b88-5206-4f22-84a6-78212e0d5f1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:43:49.835 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.833+0000 7f641bfb6640 10 mgrc shutdown 2024-04-25T19:43:49.835 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.834+0000 7f641bfb6640 1 -- 172.21.15.29:0/212426973 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f64140a92b0 msgr2=0x7f64140af300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:49.835 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.834+0000 7f641bfb6640 1 --2- 172.21.15.29:0/212426973 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f64140a92b0 0x7f64140af300 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6404033000 tx=0x7f64040046c0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.835 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.834+0000 7f641bfb6640 1 -- 172.21.15.29:0/212426973 shutdown_connections 2024-04-25T19:43:49.835 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.834+0000 7f641bfb6640 1 --2- 172.21.15.29:0/212426973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64140ad120 0x7f64141dd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.835 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.834+0000 7f641bfb6640 1 --2- 172.21.15.29:0/212426973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64140aa1f0 0x7f64141dd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.835 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.834+0000 7f641bfb6640 1 --2- 172.21.15.29:0/212426973 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f64140a92b0 0x7f64140af300 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:49.835 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.834+0000 7f641bfb6640 1 -- 172.21.15.29:0/212426973 >> 172.21.15.29:0/212426973 conn(0x7f641406d6a0 msgr2=0x7f6414146f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:43:49.835 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.834+0000 7f641bfb6640 1 -- 172.21.15.29:0/212426973 shutdown_connections 2024-04-25T19:43:49.835 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:49.834+0000 7f641bfb6640 1 -- 172.21.15.29:0/212426973 wait complete. 2024-04-25T19:43:49.844 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-04-25T19:43:49.895 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:49.894+0000 7ffac5ec7040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:43:50.217 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:50.216+0000 7ff865a32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T19:43:50.218 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:50.216+0000 7fc87723c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T19:43:50.298 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:50.297+0000 7ffac5ec7040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T19:43:50.483 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:50.481+0000 7ff865a32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T19:43:50.483 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:50.482+0000 7fc87723c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T19:43:50.538 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:50.536+0000 7ff865a32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T19:43:50.539 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:50.537+0000 7fc87723c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T19:43:50.572 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:50.571+0000 7ffac5ec7040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T19:43:50.627 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:50.626+0000 7ffac5ec7040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T19:43:50.655 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:50.653+0000 7ff865a32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T19:43:50.655 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:50.653+0000 7fc87723c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T19:43:50.751 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:50.750+0000 7ffac5ec7040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T19:43:50.939 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:50.938+0000 7ff865a32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T19:43:50.940 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:50.939+0000 7fc87723c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T19:43:50.987 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:50.985+0000 7ff865a32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T19:43:50.988 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:50.987+0000 7fc87723c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T19:43:51.046 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:51.045+0000 7ffac5ec7040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T19:43:51.096 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:51.095+0000 7ffac5ec7040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T19:43:51.148 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:51.147+0000 7ff865a32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T19:43:51.149 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:51.148+0000 7fc87723c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T19:43:51.261 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:51.260+0000 7ffac5ec7040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T19:43:51.287 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:51.285+0000 7ff865a32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T19:43:51.338 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:51.337+0000 7ff865a32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T19:43:51.345 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:51.343+0000 7fc87723c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T19:43:51.396 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:51.394+0000 7fc87723c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T19:43:51.406 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:51.405+0000 7ffac5ec7040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T19:43:51.411 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:51.409+0000 7ff865a32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T19:43:51.453 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:51.451+0000 7fc87723c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T19:43:51.458 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:51.457+0000 7ff865a32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T19:43:51.459 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:51.458+0000 7ffac5ec7040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T19:43:51.497 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:51.496+0000 7fc87723c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T19:43:51.533 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:51.532+0000 7ffac5ec7040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T19:43:51.580 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:51.579+0000 7ffac5ec7040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T19:43:51.951 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:51.950+0000 7ff865a32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T19:43:51.952 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:51.950+0000 7fc87723c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T19:43:52.071 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:52.070+0000 7ff865a32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T19:43:52.072 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:43:52.071+0000 7fc87723c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T19:43:52.086 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:52.085+0000 7ffac5ec7040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T19:43:52.215 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:43:52.214+0000 7ffac5ec7040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T19:43:53.092 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:53.089+0000 7ff80551b640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-25T19:43:53.092 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:43:53.090+0000 7ff80551b640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-25T19:43:53.138 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-04-25T19:43:53.137+0000 7f4ab0843640 -1 osd.0 0 log_to_monitors true 2024-04-25T19:43:53.185 INFO:tasks.ceph.osd.1.smithi102.stderr:2024-04-25T19:43:53.183+0000 7f635c101640 -1 osd.1 0 log_to_monitors true 2024-04-25T19:43:53.230 INFO:tasks.ceph.osd.3.smithi102.stderr:2024-04-25T19:43:53.228+0000 7f054dc12640 -1 osd.3 0 log_to_monitors true 2024-04-25T19:43:53.239 INFO:tasks.ceph.osd.2.smithi102.stderr:2024-04-25T19:43:53.237+0000 7f413c01e640 -1 osd.2 0 log_to_monitors true 2024-04-25T19:43:55.759 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-04-25T19:43:55.758+0000 7f4aacfed640 -1 osd.0 0 waiting for initial osdmap 2024-04-25T19:43:55.778 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-04-25T19:43:55.777+0000 7f4aa45d4640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T19:43:56.248 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T19:43:56.356 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.355+0000 7f922aef9640 1 Processor -- start 2024-04-25T19:43:56.356 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.355+0000 7f922aef9640 1 -- start start 2024-04-25T19:43:56.358 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.356+0000 7f922aef9640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f92240aa6f0 0x7f9224146210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:56.358 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.356+0000 7f922aef9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9224146750 0x7f9224151450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:56.359 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.356+0000 7f922aef9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92240ad2f0 con 0x7f922406e1e0 2024-04-25T19:43:56.359 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.356+0000 7f922aef9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f92240aad80 con 0x7f9224146750 2024-04-25T19:43:56.359 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.356+0000 7f922aef9640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f92240ad5b0 con 0x7f92240aa6f0 2024-04-25T19:43:56.359 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.356+0000 7f9223fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f92240aa6f0 0x7f9224146210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:56.359 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.356+0000 7f9229496640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9224146750 0x7f9224151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:56.359 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.356+0000 7f9223fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f92240aa6f0 0x7f9224146210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:43336/0 (socket says 172.21.15.29:43336) 2024-04-25T19:43:56.359 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.356+0000 7f9228c95640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f922406e1e0 0x7f92240ac690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58648/0 (socket says 172.21.15.29:58648) 2024-04-25T19:43:56.359 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.356+0000 7f9223fff640 1 -- 172.21.15.29:0/2065921447 learned_addr learned my addr 172.21.15.29:0/2065921447 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:43:56.359 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.356+0000 7f9223fff640 1 -- 172.21.15.29:0/2065921447 >> v2:172.21.15.29:3300/0 conn(0x7f9224146750 msgr2=0x7f9224151450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:43:56.359 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.356+0000 7f9223fff640 1 --2- 172.21.15.29:0/2065921447 >> v2:172.21.15.29:3300/0 conn(0x7f9224146750 0x7f9224151450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:56.359 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.356+0000 7f9223fff640 1 -- 172.21.15.29:0/2065921447 >> v1:172.21.15.29:6789/0 conn(0x7f922406e1e0 legacy=0x7f92240ac690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:56.359 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.356+0000 7f9223fff640 1 -- 172.21.15.29:0/2065921447 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92240aab00 con 0x7f92240aa6f0 2024-04-25T19:43:56.359 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.356+0000 7f9229496640 1 --2- 172.21.15.29:0/2065921447 >> v2:172.21.15.29:3300/0 conn(0x7f9224146750 0x7f9224151450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:43:56.359 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.357+0000 7f9223fff640 1 --2- 172.21.15.29:0/2065921447 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f92240aa6f0 0x7f9224146210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9210009870 tx=0x7f921002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d2fd870cb97b970 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:43:56.360 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.357+0000 7f92237fe640 1 -- 172.21.15.29:0/2065921447 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9210035020 con 0x7f92240aa6f0 2024-04-25T19:43:56.360 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.357+0000 7f92237fe640 1 -- 172.21.15.29:0/2065921447 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9210031640 con 0x7f92240aa6f0 2024-04-25T19:43:56.360 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.357+0000 7f92237fe640 1 -- 172.21.15.29:0/2065921447 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f921002d040 con 0x7f92240aa6f0 2024-04-25T19:43:56.361 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.358+0000 7f922aef9640 1 -- 172.21.15.29:0/2065921447 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f92240aa6f0 msgr2=0x7f9224146210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:56.361 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.358+0000 7f922aef9640 1 --2- 172.21.15.29:0/2065921447 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f92240aa6f0 0x7f9224146210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9210009870 tx=0x7f921002b8f0 comp rx=0 tx=0).stop 2024-04-25T19:43:56.361 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.358+0000 7f922aef9640 1 -- 172.21.15.29:0/2065921447 shutdown_connections 2024-04-25T19:43:56.361 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.358+0000 7f922aef9640 1 --2- 172.21.15.29:0/2065921447 >> v2:172.21.15.29:3300/0 conn(0x7f9224146750 0x7f9224151450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:56.361 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.358+0000 7f922aef9640 1 --2- 172.21.15.29:0/2065921447 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f92240aa6f0 0x7f9224146210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:56.361 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.358+0000 7f922aef9640 1 -- 172.21.15.29:0/2065921447 >> 172.21.15.29:0/2065921447 conn(0x7f92240ab710 msgr2=0x7f92240bbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:43:56.362 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.358+0000 7f922aef9640 1 -- 172.21.15.29:0/2065921447 shutdown_connections 2024-04-25T19:43:56.362 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.358+0000 7f922aef9640 1 -- 172.21.15.29:0/2065921447 wait complete. 2024-04-25T19:43:56.362 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.360+0000 7f922aef9640 1 Processor -- start 2024-04-25T19:43:56.362 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.360+0000 7f922aef9640 1 -- start start 2024-04-25T19:43:56.364 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.361+0000 7f922aef9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f922406e1e0 0x7f922413b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:56.364 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.361+0000 7f922aef9640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92240aa6f0 0x7f9224140700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:56.364 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.361+0000 7f922aef9640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9224146750 0x7f9224140c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:56.364 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.361+0000 7f922aef9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9224151b10 con 0x7f922406e1e0 2024-04-25T19:43:56.364 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.361+0000 7f922aef9640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9224151990 con 0x7f92240aa6f0 2024-04-25T19:43:56.364 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.361+0000 7f922aef9640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f9224151c90 con 0x7f9224146750 2024-04-25T19:43:56.364 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.361+0000 7f9228c95640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f922406e1e0 0x7f922413b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:56.364 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.361+0000 7f9228c95640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f922406e1e0 0x7f922413b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58246/0 (socket says 172.21.15.29:58246) 2024-04-25T19:43:56.364 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.361+0000 7f9228c95640 1 -- 172.21.15.29:0/549722383 learned_addr learned my addr 172.21.15.29:0/549722383 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:43:56.365 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.361+0000 7f9229496640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9224146750 0x7f9224140c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:56.365 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.361+0000 7f9223fff640 1 --2- 172.21.15.29:0/549722383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92240aa6f0 0x7f9224140700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:56.365 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.361+0000 7f9229496640 1 -- 172.21.15.29:0/549722383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92240aa6f0 msgr2=0x7f9224140700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:56.365 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.361+0000 7f9229496640 1 --2- 172.21.15.29:0/549722383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92240aa6f0 0x7f9224140700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:56.365 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.361+0000 7f9229496640 1 -- 172.21.15.29:0/549722383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f922406e1e0 msgr2=0x7f922413b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:56.365 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.361+0000 7f9229496640 1 --2- 172.21.15.29:0/549722383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f922406e1e0 0x7f922413b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:56.365 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.361+0000 7f9229496640 1 -- 172.21.15.29:0/549722383 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92241412e0 con 0x7f9224146750 2024-04-25T19:43:56.365 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.361+0000 7f9228c95640 1 --2- 172.21.15.29:0/549722383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f922406e1e0 0x7f922413b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:43:56.365 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.361+0000 7f9229496640 1 --2- 172.21.15.29:0/549722383 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9224146750 0x7f9224140c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f921c00ef90 tx=0x7f921c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:43:56.365 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.361+0000 7f92217fa640 1 -- 172.21.15.29:0/549722383 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f921c018030 con 0x7f9224146750 2024-04-25T19:43:56.365 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.361+0000 7f92217fa640 1 -- 172.21.15.29:0/549722383 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f921c00cd30 con 0x7f9224146750 2024-04-25T19:43:56.365 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.362+0000 7f922aef9640 1 -- 172.21.15.29:0/549722383 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92240b96a0 con 0x7f9224146750 2024-04-25T19:43:56.366 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.362+0000 7f92217fa640 1 -- 172.21.15.29:0/549722383 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f921c003bb0 con 0x7f9224146750 2024-04-25T19:43:56.366 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.362+0000 7f922aef9640 1 -- 172.21.15.29:0/549722383 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9224141800 con 0x7f9224146750 2024-04-25T19:43:56.366 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.362+0000 7f922aef9640 1 -- 172.21.15.29:0/549722383 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9224050740 con 0x7f9224146750 2024-04-25T19:43:56.367 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.365+0000 7f92217fa640 1 -- 172.21.15.29:0/549722383 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 4) ==== 144803+0+0 (secure 0 0 0) 0x7f921c005d10 con 0x7f9224146750 2024-04-25T19:43:56.367 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.365+0000 7f92217fa640 20 mgrc handle_mgr_map mgrmap(e 4) 2024-04-25T19:43:56.367 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.365+0000 7f92217fa640 4 mgrc handle_mgr_map Got map version 4 2024-04-25T19:43:56.367 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.365+0000 7f92217fa640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:43:56.367 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.365+0000 7f92217fa640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:43:56.367 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.365+0000 7f92217fa640 1 --2- 172.21.15.29:0/549722383 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f92080ac4a0 0x7f92080ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:43:56.367 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.366+0000 7f92217fa640 1 -- 172.21.15.29:0/549722383 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1903+0+0 (secure 0 0 0) 0x7f921c0d9ce0 con 0x7f9224146750 2024-04-25T19:43:56.367 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.366+0000 7f9228c95640 1 --2- 172.21.15.29:0/549722383 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f92080ac4a0 0x7f92080ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:43:56.367 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.366+0000 7f9228c95640 1 --2- 172.21.15.29:0/549722383 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f92080ac4a0 0x7f92080ae9b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9218004e00 tx=0x7f9218007f10 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:43:56.370 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.369+0000 7f92217fa640 1 -- 172.21.15.29:0/549722383 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f921c022070 con 0x7f9224146750 2024-04-25T19:43:56.553 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.552+0000 7f922aef9640 1 -- 172.21.15.29:0/549722383 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f92240aa4d0 con 0x7f9224146750 2024-04-25T19:43:56.554 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.553+0000 7f92217fa640 1 -- 172.21.15.29:0/549722383 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v9) ==== 74+0+4637 (secure 0 0 0) 0x7f921c010040 con 0x7f9224146750 2024-04-25T19:43:56.554 INFO:teuthology.misc.health.smithi029.stdout: 2024-04-25T19:43:56.554 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":9,"fsid":"1b3c39b5-e787-409e-b0bd-f4f491c89d72","created":"2024-04-25T19:43:47.854697+0000","modified":"2024-04-25T19:43:56.140656+0000","last_up_change":"2024-04-25T19:43:56.140656+0000","last_in_change":"2024-04-25T19:43:49.022776+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4d805223-ee84-494f-bd31-59091906aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6801","nonce":695704147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6803","nonce":695704147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6807","nonce":695704147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6805","nonce":695704147}]},"public_addr":"172.21.15.29:6801/695704147","cluster_addr":"172.21.15.29:6803/695704147","heartbeat_back_addr":"172.21.15.29:6807/695704147","heartbeat_front_addr":"172.21.15.29:6805/695704147","state":["exists","up"]},{"osd":1,"uuid":"607c5038-38af-440f-8371-e71758549920","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6859ae76-54bb-4c55-b0dc-5e2735c0d31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a98b7b88-5206-4f22-84a6-78212e0d5f1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:43:56.556 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.555+0000 7f922aef9640 10 mgrc shutdown 2024-04-25T19:43:56.557 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.555+0000 7f922aef9640 1 -- 172.21.15.29:0/549722383 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f92080ac4a0 msgr2=0x7f92080ae9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:56.557 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.555+0000 7f922aef9640 1 --2- 172.21.15.29:0/549722383 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f92080ac4a0 0x7f92080ae9b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9218004e00 tx=0x7f9218007f10 comp rx=0 tx=0).stop 2024-04-25T19:43:56.557 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.555+0000 7f922aef9640 1 -- 172.21.15.29:0/549722383 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9224146750 msgr2=0x7f9224140c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:43:56.557 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.555+0000 7f922aef9640 1 --2- 172.21.15.29:0/549722383 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9224146750 0x7f9224140c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f921c00ef90 tx=0x7f921c00c490 comp rx=0 tx=0).stop 2024-04-25T19:43:56.557 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.555+0000 7f922aef9640 1 -- 172.21.15.29:0/549722383 shutdown_connections 2024-04-25T19:43:56.557 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.555+0000 7f922aef9640 1 --2- 172.21.15.29:0/549722383 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f92080ac4a0 0x7f92080ae9b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:56.557 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.555+0000 7f922aef9640 1 --2- 172.21.15.29:0/549722383 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9224146750 0x7f9224140c40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:56.557 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.555+0000 7f922aef9640 1 --2- 172.21.15.29:0/549722383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92240aa6f0 0x7f9224140700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:56.557 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.555+0000 7f922aef9640 1 --2- 172.21.15.29:0/549722383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f922406e1e0 0x7f922413b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:43:56.557 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.555+0000 7f922aef9640 1 -- 172.21.15.29:0/549722383 >> 172.21.15.29:0/549722383 conn(0x7f92240ab710 msgr2=0x7f92241430c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:43:56.557 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.555+0000 7f922aef9640 1 -- 172.21.15.29:0/549722383 shutdown_connections 2024-04-25T19:43:56.557 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:43:56.555+0000 7f922aef9640 1 -- 172.21.15.29:0/549722383 wait complete. 2024-04-25T19:43:56.566 DEBUG:teuthology.misc:1 of 4 OSDs are up 2024-04-25T19:43:56.851 INFO:tasks.ceph.osd.1.smithi102.stderr:2024-04-25T19:43:56.849+0000 7f6358097640 -1 osd.1 0 waiting for initial osdmap 2024-04-25T19:43:56.866 INFO:tasks.ceph.osd.1.smithi102.stderr:2024-04-25T19:43:56.865+0000 7f634fe92640 -1 osd.1 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T19:43:56.874 INFO:tasks.ceph.osd.3.smithi102.stderr:2024-04-25T19:43:56.873+0000 7f0549554640 -1 osd.3 0 waiting for initial osdmap 2024-04-25T19:43:56.887 INFO:tasks.ceph.osd.2.smithi102.stderr:2024-04-25T19:43:56.886+0000 7f41378e8640 -1 osd.2 0 waiting for initial osdmap 2024-04-25T19:43:56.888 INFO:tasks.ceph.osd.3.smithi102.stderr:2024-04-25T19:43:56.887+0000 7f054134f640 -1 osd.3 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T19:43:56.896 INFO:tasks.ceph.osd.2.smithi102.stderr:2024-04-25T19:43:56.895+0000 7f412f6f4640 -1 osd.2 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T19:43:58.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:43:58.055 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:44:02.971 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T19:44:03.087 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.086+0000 7fbd5618e640 1 Processor -- start 2024-04-25T19:44:03.087 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.086+0000 7fbd5618e640 1 -- start start 2024-04-25T19:44:03.089 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.087+0000 7fbd5618e640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbd5014e1d0 0x7fbd501505f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:03.089 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.087+0000 7fbd5618e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd50137cd0 con 0x7fbd50137040 2024-04-25T19:44:03.089 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.087+0000 7fbd5618e640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd500b7800 con 0x7fbd500addb0 2024-04-25T19:44:03.089 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.087+0000 7fbd5618e640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fbd500ae1c0 con 0x7fbd5014e1d0 2024-04-25T19:44:03.089 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.087+0000 7fbd4ffff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbd5014e1d0 0x7fbd501505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:03.089 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.087+0000 7fbd47fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbd50137040 0x7fbd50137450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55382/0 (socket says 172.21.15.29:55382) 2024-04-25T19:44:03.089 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.087+0000 7fbd47fff640 1 -- 172.21.15.29:0/2975802067 learned_addr learned my addr 172.21.15.29:0/2975802067 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:03.089 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.088+0000 7fbd4effd640 1 -- 172.21.15.29:0/2975802067 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2182358125 0 0) 0x7fbd50137cd0 con 0x7fbd50137040 2024-04-25T19:44:03.089 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.088+0000 7fbd4effd640 1 -- 172.21.15.29:0/2975802067 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd3c003520 con 0x7fbd50137040 2024-04-25T19:44:03.089 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.088+0000 7fbd4ffff640 1 -- 172.21.15.29:0/2975802067 >> v1:172.21.15.102:6789/0 conn(0x7fbd500addb0 legacy=0x7fbd500b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:03.089 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.088+0000 7fbd4ffff640 1 -- 172.21.15.29:0/2975802067 >> v1:172.21.15.29:6789/0 conn(0x7fbd50137040 legacy=0x7fbd50137450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:03.090 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.088+0000 7fbd4ffff640 1 -- 172.21.15.29:0/2975802067 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd500ae340 con 0x7fbd5014e1d0 2024-04-25T19:44:03.091 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.088+0000 7fbd4ffff640 1 --2- 172.21.15.29:0/2975802067 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbd5014e1d0 0x7fbd501505f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbd48009dc0 tx=0x7fbd4802be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebd2f0cd1c36464a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:03.091 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.089+0000 7fbd4effd640 1 -- 172.21.15.29:0/2975802067 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd48039070 con 0x7fbd5014e1d0 2024-04-25T19:44:03.091 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.089+0000 7fbd4effd640 1 -- 172.21.15.29:0/2975802067 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd4802cc40 con 0x7fbd5014e1d0 2024-04-25T19:44:03.091 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.089+0000 7fbd4effd640 1 -- 172.21.15.29:0/2975802067 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd48034440 con 0x7fbd5014e1d0 2024-04-25T19:44:03.091 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.089+0000 7fbd5618e640 1 -- 172.21.15.29:0/2975802067 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbd5014e1d0 msgr2=0x7fbd501505f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:03.091 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.089+0000 7fbd5618e640 1 --2- 172.21.15.29:0/2975802067 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbd5014e1d0 0x7fbd501505f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbd48009dc0 tx=0x7fbd4802be90 comp rx=0 tx=0).stop 2024-04-25T19:44:03.091 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.089+0000 7fbd5618e640 1 -- 172.21.15.29:0/2975802067 shutdown_connections 2024-04-25T19:44:03.091 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.089+0000 7fbd5618e640 1 --2- 172.21.15.29:0/2975802067 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbd5014e1d0 0x7fbd501505f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:03.091 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.089+0000 7fbd5618e640 1 -- 172.21.15.29:0/2975802067 >> 172.21.15.29:0/2975802067 conn(0x7fbd50072b20 msgr2=0x7fbd500a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:03.091 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.089+0000 7fbd5618e640 1 -- 172.21.15.29:0/2975802067 shutdown_connections 2024-04-25T19:44:03.091 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.090+0000 7fbd5618e640 1 -- 172.21.15.29:0/2975802067 wait complete. 2024-04-25T19:44:03.092 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.091+0000 7fbd5618e640 1 Processor -- start 2024-04-25T19:44:03.092 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.091+0000 7fbd5618e640 1 -- start start 2024-04-25T19:44:03.094 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.092+0000 7fbd5618e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd500addb0 0x7fbd500b9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:03.094 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.092+0000 7fbd5618e640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbd50137040 0x7fbd5014ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:03.094 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.092+0000 7fbd5618e640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd5014e1d0 0x7fbd50133390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:03.094 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.092+0000 7fbd5618e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbd50150b30 con 0x7fbd500addb0 2024-04-25T19:44:03.095 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.092+0000 7fbd5618e640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5013b1c0 con 0x7fbd5014e1d0 2024-04-25T19:44:03.095 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.092+0000 7fbd5618e640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fbd50150cb0 con 0x7fbd50137040 2024-04-25T19:44:03.096 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.092+0000 7fbd47fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbd50137040 0x7fbd5014ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:03.096 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.092+0000 7fbd4f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd500addb0 0x7fbd500b9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:03.096 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.092+0000 7fbd47fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbd50137040 0x7fbd5014ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:60176/0 (socket says 172.21.15.29:60176) 2024-04-25T19:44:03.096 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.092+0000 7fbd47fff640 1 -- 172.21.15.29:0/242601694 learned_addr learned my addr 172.21.15.29:0/242601694 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:03.096 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.092+0000 7fbd4ffff640 1 --2- 172.21.15.29:0/242601694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd5014e1d0 0x7fbd50133390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:03.096 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.092+0000 7fbd47fff640 1 -- 172.21.15.29:0/242601694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd5014e1d0 msgr2=0x7fbd50133390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:03.096 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.092+0000 7fbd47fff640 1 --2- 172.21.15.29:0/242601694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd5014e1d0 0x7fbd50133390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:03.096 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.092+0000 7fbd47fff640 1 -- 172.21.15.29:0/242601694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd500addb0 msgr2=0x7fbd500b9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:03.096 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.092+0000 7fbd47fff640 1 --2- 172.21.15.29:0/242601694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd500addb0 0x7fbd500b9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:03.096 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.092+0000 7fbd47fff640 1 -- 172.21.15.29:0/242601694 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd5014f7e0 con 0x7fbd50137040 2024-04-25T19:44:03.096 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.092+0000 7fbd4ffff640 1 --2- 172.21.15.29:0/242601694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd5014e1d0 0x7fbd50133390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:44:03.096 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.093+0000 7fbd47fff640 1 --2- 172.21.15.29:0/242601694 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbd50137040 0x7fbd5014ff90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbd3800c3e0 tx=0x7fbd3800c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:03.096 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.093+0000 7fbd4cff9640 1 -- 172.21.15.29:0/242601694 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd38017070 con 0x7fbd50137040 2024-04-25T19:44:03.096 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.093+0000 7fbd4cff9640 1 -- 172.21.15.29:0/242601694 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd38005ce0 con 0x7fbd50137040 2024-04-25T19:44:03.096 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.093+0000 7fbd5618e640 1 -- 172.21.15.29:0/242601694 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd500b97c0 con 0x7fbd50137040 2024-04-25T19:44:03.097 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.093+0000 7fbd4cff9640 1 -- 172.21.15.29:0/242601694 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd380072d0 con 0x7fbd50137040 2024-04-25T19:44:03.097 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.093+0000 7fbd5618e640 1 -- 172.21.15.29:0/242601694 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd5014ec30 con 0x7fbd50137040 2024-04-25T19:44:03.098 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.094+0000 7fbd5618e640 1 -- 172.21.15.29:0/242601694 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd5014d500 con 0x7fbd50137040 2024-04-25T19:44:03.098 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.095+0000 7fbd4cff9640 1 -- 172.21.15.29:0/242601694 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7fbd3801a070 con 0x7fbd50137040 2024-04-25T19:44:03.098 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.095+0000 7fbd4cff9640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:03.098 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.096+0000 7fbd4cff9640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:03.098 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.096+0000 7fbd4cff9640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:03.098 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.096+0000 7fbd4cff9640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:03.098 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.096+0000 7fbd4cff9640 1 --2- 172.21.15.29:0/242601694 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fbd300ac530 0x7fbd300aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:03.098 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.096+0000 7fbd4cff9640 1 -- 172.21.15.29:0/242601694 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(13..13 src has 1..13) ==== 3129+0+0 (secure 0 0 0) 0x7fbd380d8610 con 0x7fbd50137040 2024-04-25T19:44:03.098 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.096+0000 7fbd4f7fe640 1 --2- 172.21.15.29:0/242601694 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fbd300ac530 0x7fbd300aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:03.098 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.097+0000 7fbd4f7fe640 1 --2- 172.21.15.29:0/242601694 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fbd300ac530 0x7fbd300aea40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbd400016e0 tx=0x7fbd40009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:03.100 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.100+0000 7fbd4cff9640 1 -- 172.21.15.29:0/242601694 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbd380a30e0 con 0x7fbd50137040 2024-04-25T19:44:03.291 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.290+0000 7fbd5618e640 1 -- 172.21.15.29:0/242601694 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbd500aa9a0 con 0x7fbd50137040 2024-04-25T19:44:03.292 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.291+0000 7fbd4cff9640 1 -- 172.21.15.29:0/242601694 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+7966 (secure 0 0 0) 0x7fbd380a73e0 con 0x7fbd50137040 2024-04-25T19:44:03.292 INFO:teuthology.misc.health.smithi029.stdout: 2024-04-25T19:44:03.292 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":13,"fsid":"1b3c39b5-e787-409e-b0bd-f4f491c89d72","created":"2024-04-25T19:43:47.854697+0000","modified":"2024-04-25T19:44:00.866320+0000","last_up_change":"2024-04-25T19:43:57.853237+0000","last_in_change":"2024-04-25T19:43:49.022776+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:43:58.096700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d805223-ee84-494f-bd31-59091906aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6801","nonce":695704147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6803","nonce":695704147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6807","nonce":695704147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6805","nonce":695704147}]},"public_addr":"172.21.15.29:6801/695704147","cluster_addr":"172.21.15.29:6803/695704147","heartbeat_back_addr":"172.21.15.29:6807/695704147","heartbeat_front_addr":"172.21.15.29:6805/695704147","state":["exists","up"]},{"osd":1,"uuid":"607c5038-38af-440f-8371-e71758549920","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6814","nonce":3135821452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6817","nonce":3135821452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6823","nonce":3135821452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6821","nonce":3135821452}]},"public_addr":"172.21.15.102:6814/3135821452","cluster_addr":"172.21.15.102:6817/3135821452","heartbeat_back_addr":"172.21.15.102:6823/3135821452","heartbeat_front_addr":"172.21.15.102:6821/3135821452","state":["exists","up"]},{"osd":2,"uuid":"6859ae76-54bb-4c55-b0dc-5e2735c0d31c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6801","nonce":2071844144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6803","nonce":2071844144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6811","nonce":2071844144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6806","nonce":2071844144}]},"public_addr":"172.21.15.102:6801/2071844144","cluster_addr":"172.21.15.102:6803/2071844144","heartbeat_back_addr":"172.21.15.102:6811/2071844144","heartbeat_front_addr":"172.21.15.102:6806/2071844144","state":["exists","up"]},{"osd":3,"uuid":"a98b7b88-5206-4f22-84a6-78212e0d5f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6805","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6807","nonce":339944984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6810","nonce":339944984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6820","nonce":339944984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6815","nonce":339944984}]},"public_addr":"172.21.15.102:6807/339944984","cluster_addr":"172.21.15.102:6810/339944984","heartbeat_back_addr":"172.21.15.102:6820/339944984","heartbeat_front_addr":"172.21.15.102:6815/339944984","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.121529+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.201155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.264387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.198713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:44:03.295 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.293+0000 7fbd5618e640 10 mgrc shutdown 2024-04-25T19:44:03.295 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.293+0000 7fbd5618e640 1 -- 172.21.15.29:0/242601694 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fbd300ac530 msgr2=0x7fbd300aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:03.295 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.293+0000 7fbd5618e640 1 --2- 172.21.15.29:0/242601694 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fbd300ac530 0x7fbd300aea40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbd400016e0 tx=0x7fbd40009040 comp rx=0 tx=0).stop 2024-04-25T19:44:03.295 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.293+0000 7fbd5618e640 1 -- 172.21.15.29:0/242601694 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbd50137040 msgr2=0x7fbd5014ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:03.295 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.293+0000 7fbd5618e640 1 --2- 172.21.15.29:0/242601694 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbd50137040 0x7fbd5014ff90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbd3800c3e0 tx=0x7fbd3800c8a0 comp rx=0 tx=0).stop 2024-04-25T19:44:03.295 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.293+0000 7fbd5618e640 1 -- 172.21.15.29:0/242601694 shutdown_connections 2024-04-25T19:44:03.295 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.293+0000 7fbd5618e640 1 --2- 172.21.15.29:0/242601694 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fbd300ac530 0x7fbd300aea40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:03.295 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.293+0000 7fbd5618e640 1 --2- 172.21.15.29:0/242601694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd5014e1d0 0x7fbd50133390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:03.295 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.293+0000 7fbd5618e640 1 --2- 172.21.15.29:0/242601694 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbd50137040 0x7fbd5014ff90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:03.295 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.293+0000 7fbd5618e640 1 --2- 172.21.15.29:0/242601694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd500addb0 0x7fbd500b9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:03.295 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.293+0000 7fbd5618e640 1 -- 172.21.15.29:0/242601694 >> 172.21.15.29:0/242601694 conn(0x7fbd50072b20 msgr2=0x7fbd5006cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:03.295 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.293+0000 7fbd5618e640 1 -- 172.21.15.29:0/242601694 shutdown_connections 2024-04-25T19:44:03.295 INFO:teuthology.misc.health.smithi029.stderr:2024-04-25T19:44:03.293+0000 7fbd5618e640 1 -- 172.21.15.29:0/242601694 wait complete. 2024-04-25T19:44:03.304 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-04-25T19:44:03.304 INFO:tasks.ceph:Creating RBD pool 2024-04-25T19:44:03.304 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-25T19:44:03.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.413+0000 7f721a743640 1 Processor -- start 2024-04-25T19:44:03.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.413+0000 7f721a743640 1 -- start start 2024-04-25T19:44:03.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.414+0000 7f721a743640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f721406df20 0x7f7214073920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:03.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.414+0000 7f721a743640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f72141465a0 0x7f721414a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:03.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.414+0000 7f721a743640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72140a6e30 con 0x7f72141469b0 2024-04-25T19:44:03.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.414+0000 7f721a743640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f721406e330 con 0x7f72141465a0 2024-04-25T19:44:03.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.414+0000 7f721a743640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f721414b1a0 con 0x7f721406df20 2024-04-25T19:44:03.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.414+0000 7f7213fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f721406df20 0x7f7214073920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:03.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.414+0000 7f7218ce0640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f72141469b0 0x7f721406ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55390/0 (socket says 172.21.15.29:55390) 2024-04-25T19:44:03.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.415+0000 7f7218ce0640 1 -- 172.21.15.29:0/2898966392 learned_addr learned my addr 172.21.15.29:0/2898966392 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:03.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.415+0000 7f72137fe640 1 --2- 172.21.15.29:0/2898966392 >> v2:172.21.15.102:3300/0 conn(0x7f72141465a0 0x7f721414a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:03.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.415+0000 7f7212ffd640 1 -- 172.21.15.29:0/2898966392 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3255262717 0 0) 0x7f720c001490 con 0x7f72141469b0 2024-04-25T19:44:03.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.415+0000 7f7213fff640 1 -- 172.21.15.29:0/2898966392 >> v2:172.21.15.102:3300/0 conn(0x7f72141465a0 msgr2=0x7f721414a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:03.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.415+0000 7f7213fff640 1 --2- 172.21.15.29:0/2898966392 >> v2:172.21.15.102:3300/0 conn(0x7f72141465a0 0x7f721414a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:03.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.415+0000 7f7213fff640 1 -- 172.21.15.29:0/2898966392 >> v1:172.21.15.29:6789/0 conn(0x7f72141469b0 legacy=0x7f721406ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:03.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.415+0000 7f7213fff640 1 -- 172.21.15.29:0/2898966392 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7214073e60 con 0x7f721406df20 2024-04-25T19:44:03.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.416+0000 7f7213fff640 1 --2- 172.21.15.29:0/2898966392 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f721406df20 0x7f7214073920 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f72080097a0 tx=0x7f720802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=286ef539f7903d8a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:03.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.416+0000 7f7212ffd640 1 -- 172.21.15.29:0/2898966392 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72080315a0 con 0x7f721406df20 2024-04-25T19:44:03.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.416+0000 7f7212ffd640 1 -- 172.21.15.29:0/2898966392 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7208037070 con 0x7f721406df20 2024-04-25T19:44:03.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.416+0000 7f7212ffd640 1 -- 172.21.15.29:0/2898966392 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7208031960 con 0x7f721406df20 2024-04-25T19:44:03.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.416+0000 7f721a743640 1 -- 172.21.15.29:0/2898966392 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f721406df20 msgr2=0x7f7214073920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:03.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.416+0000 7f721a743640 1 --2- 172.21.15.29:0/2898966392 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f721406df20 0x7f7214073920 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f72080097a0 tx=0x7f720802b840 comp rx=0 tx=0).stop 2024-04-25T19:44:03.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.416+0000 7f721a743640 1 -- 172.21.15.29:0/2898966392 shutdown_connections 2024-04-25T19:44:03.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.416+0000 7f721a743640 1 --2- 172.21.15.29:0/2898966392 >> v2:172.21.15.102:3300/0 conn(0x7f72141465a0 0x7f721414a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:03.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.416+0000 7f721a743640 1 --2- 172.21.15.29:0/2898966392 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f721406df20 0x7f7214073920 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:03.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.416+0000 7f721a743640 1 -- 172.21.15.29:0/2898966392 >> 172.21.15.29:0/2898966392 conn(0x7f72140a9390 msgr2=0x7f72140a97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:03.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.417+0000 7f721a743640 1 -- 172.21.15.29:0/2898966392 shutdown_connections 2024-04-25T19:44:03.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.417+0000 7f721a743640 1 -- 172.21.15.29:0/2898966392 wait complete. 2024-04-25T19:44:03.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.418+0000 7f721a743640 1 Processor -- start 2024-04-25T19:44:03.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.418+0000 7f721a743640 1 -- start start 2024-04-25T19:44:03.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.419+0000 7f721a743640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f721406df20 0x7f72140a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.419+0000 7f721a743640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f72141465a0 0x7f72141e0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.419+0000 7f721a743640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72141469b0 0x7f72141d2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.419+0000 7f721a743640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f721414ade0 con 0x7f721406df20 2024-04-25T19:44:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.419+0000 7f721a743640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f721414ac60 con 0x7f72141469b0 2024-04-25T19:44:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.419+0000 7f721a743640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f721414af60 con 0x7f72141465a0 2024-04-25T19:44:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.419+0000 7f72137fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f72141465a0 0x7f72141e0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.419+0000 7f7213fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f721406df20 0x7f72140a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.419+0000 7f7213fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f721406df20 0x7f72140a5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49620/0 (socket says 172.21.15.29:49620) 2024-04-25T19:44:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.419+0000 7f7213fff640 1 -- 172.21.15.29:0/2894967842 learned_addr learned my addr 172.21.15.29:0/2894967842 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.419+0000 7f72137fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f72141465a0 0x7f72141e0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:60196/0 (socket says 172.21.15.29:60196) 2024-04-25T19:44:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.419+0000 7f7218ce0640 1 --2- 172.21.15.29:0/2894967842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72141469b0 0x7f72141d2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.420+0000 7f7213fff640 1 -- 172.21.15.29:0/2894967842 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f72141465a0 msgr2=0x7f72141e0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.420+0000 7f7213fff640 1 --2- 172.21.15.29:0/2894967842 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f72141465a0 0x7f72141e0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.420+0000 7f7213fff640 1 -- 172.21.15.29:0/2894967842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72141469b0 msgr2=0x7f72141d2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:03.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.420+0000 7f7213fff640 1 --2- 172.21.15.29:0/2894967842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72141469b0 0x7f72141d2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:03.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.420+0000 7f7213fff640 1 -- 172.21.15.29:0/2894967842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f721409fc10 con 0x7f721406df20 2024-04-25T19:44:03.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.420+0000 7f72137fe640 1 --2- 172.21.15.29:0/2894967842 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f72141465a0 0x7f72141e0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:03.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.420+0000 7f7218ce0640 1 --2- 172.21.15.29:0/2894967842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72141469b0 0x7f72141d2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:44:03.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.420+0000 7f7213fff640 1 --2- 172.21.15.29:0/2894967842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f721406df20 0x7f72140a5210 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7208034040 tx=0x7f72080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:03.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.420+0000 7f7210ff9640 1 -- 172.21.15.29:0/2894967842 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7208032630 con 0x7f721406df20 2024-04-25T19:44:03.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.420+0000 7f7210ff9640 1 -- 172.21.15.29:0/2894967842 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72080327f0 con 0x7f721406df20 2024-04-25T19:44:03.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.420+0000 7f721a743640 1 -- 172.21.15.29:0/2894967842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f721409ee60 con 0x7f721406df20 2024-04-25T19:44:03.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.420+0000 7f7210ff9640 1 -- 172.21.15.29:0/2894967842 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7208032be0 con 0x7f721406df20 2024-04-25T19:44:03.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.420+0000 7f721a743640 1 -- 172.21.15.29:0/2894967842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7214145f60 con 0x7f721406df20 2024-04-25T19:44:03.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.421+0000 7f721a743640 1 -- 172.21.15.29:0/2894967842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72141e0fa0 con 0x7f721406df20 2024-04-25T19:44:03.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.422+0000 7f7210ff9640 1 -- 172.21.15.29:0/2894967842 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f720804d3f0 con 0x7f721406df20 2024-04-25T19:44:03.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.423+0000 7f7210ff9640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:03.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.423+0000 7f7210ff9640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:03.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.423+0000 7f7210ff9640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:03.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.423+0000 7f7210ff9640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:03.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.423+0000 7f7210ff9640 1 --2- 172.21.15.29:0/2894967842 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f72000ac530 0x7f72000aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:03.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.423+0000 7f7210ff9640 1 -- 172.21.15.29:0/2894967842 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 3129+0+0 (secure 0 0 0) 0x7f720802f030 con 0x7f721406df20 2024-04-25T19:44:03.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.423+0000 7f72137fe640 1 --2- 172.21.15.29:0/2894967842 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f72000ac530 0x7f72000aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:03.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.424+0000 7f72137fe640 1 --2- 172.21.15.29:0/2894967842 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f72000ac530 0x7f72000aea40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f72140a0320 tx=0x7f7204009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:03.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.427+0000 7f7210ff9640 1 -- 172.21.15.29:0/2894967842 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7208040070 con 0x7f721406df20 2024-04-25T19:44:03.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.612+0000 7f721a743640 1 -- 172.21.15.29:0/2894967842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f7214050790 con 0x7f721406df20 2024-04-25T19:44:03.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.885+0000 7f7210ff9640 1 -- 172.21.15.29:0/2894967842 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v14) ==== 109+0+0 (secure 0 0 0) 0x7f72080c5c30 con 0x7f721406df20 2024-04-25T19:44:03.886 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'rbd' created 2024-04-25T19:44:03.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.888+0000 7f721a743640 10 mgrc shutdown 2024-04-25T19:44:03.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.888+0000 7f721a743640 1 -- 172.21.15.29:0/2894967842 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f72000ac530 msgr2=0x7f72000aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:03.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.888+0000 7f721a743640 1 --2- 172.21.15.29:0/2894967842 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f72000ac530 0x7f72000aea40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f72140a0320 tx=0x7f7204009190 comp rx=0 tx=0).stop 2024-04-25T19:44:03.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.888+0000 7f721a743640 1 -- 172.21.15.29:0/2894967842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f721406df20 msgr2=0x7f72140a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:03.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.888+0000 7f721a743640 1 --2- 172.21.15.29:0/2894967842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f721406df20 0x7f72140a5210 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7208034040 tx=0x7f72080046c0 comp rx=0 tx=0).stop 2024-04-25T19:44:03.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.888+0000 7f721a743640 1 -- 172.21.15.29:0/2894967842 shutdown_connections 2024-04-25T19:44:03.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.888+0000 7f721a743640 1 --2- 172.21.15.29:0/2894967842 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f72000ac530 0x7f72000aea40 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:03.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.888+0000 7f721a743640 1 --2- 172.21.15.29:0/2894967842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72141469b0 0x7f72141d2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:03.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.888+0000 7f721a743640 1 --2- 172.21.15.29:0/2894967842 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f72141465a0 0x7f72141e0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:03.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.888+0000 7f721a743640 1 --2- 172.21.15.29:0/2894967842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f721406df20 0x7f72140a5210 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:03.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.888+0000 7f721a743640 1 -- 172.21.15.29:0/2894967842 >> 172.21.15.29:0/2894967842 conn(0x7f72140a9390 msgr2=0x7f7214070740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:03.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.888+0000 7f721a743640 1 -- 172.21.15.29:0/2894967842 shutdown_connections 2024-04-25T19:44:03.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:03.888+0000 7f721a743640 1 -- 172.21.15.29:0/2894967842 wait complete. 2024-04-25T19:44:03.901 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-25T19:44:04.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.014+0000 7feb472a7640 1 Processor -- start 2024-04-25T19:44:04.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.014+0000 7feb472a7640 1 -- start start 2024-04-25T19:44:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.015+0000 7feb472a7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feb400aa8f0 0x7feb400acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.015+0000 7feb472a7640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feb400ad330 0x7feb40146210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.015+0000 7feb472a7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb400a5e60 con 0x7feb40146750 2024-04-25T19:44:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.015+0000 7feb472a7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7feb40146b60 con 0x7feb400aa8f0 2024-04-25T19:44:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.015+0000 7feb472a7640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7feb400adaa0 con 0x7feb400ad330 2024-04-25T19:44:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.015+0000 7feb44842640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feb400ad330 0x7feb40146210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.015+0000 7feb45043640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feb400aa8f0 0x7feb400acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.015+0000 7feb44842640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feb400ad330 0x7feb40146210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:60200/0 (socket says 172.21.15.29:60200) 2024-04-25T19:44:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.015+0000 7feb44842640 1 -- 172.21.15.29:0/2740941431 learned_addr learned my addr 172.21.15.29:0/2740941431 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.016+0000 7feb44842640 1 -- 172.21.15.29:0/2740941431 >> v2:172.21.15.29:3300/0 conn(0x7feb400aa8f0 msgr2=0x7feb400acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:04.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.016+0000 7feb44842640 1 --2- 172.21.15.29:0/2740941431 >> v2:172.21.15.29:3300/0 conn(0x7feb400aa8f0 0x7feb400acdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:04.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.016+0000 7feb44842640 1 -- 172.21.15.29:0/2740941431 >> v1:172.21.15.29:6789/0 conn(0x7feb40146750 legacy=0x7feb4014a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:04.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.016+0000 7feb44842640 1 -- 172.21.15.29:0/2740941431 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb400ad7e0 con 0x7feb400ad330 2024-04-25T19:44:04.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.016+0000 7feb45043640 1 --2- 172.21.15.29:0/2740941431 >> v2:172.21.15.29:3300/0 conn(0x7feb400aa8f0 0x7feb400acdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:04.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.016+0000 7feb44842640 1 --2- 172.21.15.29:0/2740941431 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feb400ad330 0x7feb40146210 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7feb34009870 tx=0x7feb3402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3606c9cb973a320a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:04.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.016+0000 7feb3ffff640 1 -- 172.21.15.29:0/2740941431 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb34034070 con 0x7feb400ad330 2024-04-25T19:44:04.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.017+0000 7feb3ffff640 1 -- 172.21.15.29:0/2740941431 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb34030840 con 0x7feb400ad330 2024-04-25T19:44:04.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.017+0000 7feb3ffff640 1 -- 172.21.15.29:0/2740941431 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb34030c10 con 0x7feb400ad330 2024-04-25T19:44:04.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.017+0000 7feb472a7640 1 -- 172.21.15.29:0/2740941431 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feb400ad330 msgr2=0x7feb40146210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:04.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.017+0000 7feb472a7640 1 --2- 172.21.15.29:0/2740941431 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feb400ad330 0x7feb40146210 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7feb34009870 tx=0x7feb3402b980 comp rx=0 tx=0).stop 2024-04-25T19:44:04.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.017+0000 7feb472a7640 1 -- 172.21.15.29:0/2740941431 shutdown_connections 2024-04-25T19:44:04.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.017+0000 7feb472a7640 1 --2- 172.21.15.29:0/2740941431 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feb400ad330 0x7feb40146210 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:04.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.017+0000 7feb472a7640 1 --2- 172.21.15.29:0/2740941431 >> v2:172.21.15.29:3300/0 conn(0x7feb400aa8f0 0x7feb400acdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:04.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.017+0000 7feb472a7640 1 -- 172.21.15.29:0/2740941431 >> 172.21.15.29:0/2740941431 conn(0x7feb400ab2c0 msgr2=0x7feb400ab6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:04.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.017+0000 7feb472a7640 1 -- 172.21.15.29:0/2740941431 shutdown_connections 2024-04-25T19:44:04.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.017+0000 7feb472a7640 1 -- 172.21.15.29:0/2740941431 wait complete. 2024-04-25T19:44:04.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.019+0000 7feb472a7640 1 Processor -- start 2024-04-25T19:44:04.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.019+0000 7feb472a7640 1 -- start start 2024-04-25T19:44:04.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.020+0000 7feb472a7640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feb400aa8f0 0x7feb4014a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.020+0000 7feb472a7640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb400ad330 0x7feb40139e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.020+0000 7feb472a7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb40146750 0x7feb401e0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.020+0000 7feb472a7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feb4006d310 con 0x7feb40146750 2024-04-25T19:44:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.020+0000 7feb472a7640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7feb4006ceb0 con 0x7feb400ad330 2024-04-25T19:44:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.020+0000 7feb472a7640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7feb401e05d0 con 0x7feb400aa8f0 2024-04-25T19:44:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.020+0000 7feb45043640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feb400aa8f0 0x7feb4014a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.020+0000 7feb45043640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feb400aa8f0 0x7feb4014a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:60206/0 (socket says 172.21.15.29:60206) 2024-04-25T19:44:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.020+0000 7feb45043640 1 -- 172.21.15.29:0/1191506616 learned_addr learned my addr 172.21.15.29:0/1191506616 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.020+0000 7feb45844640 1 --2- 172.21.15.29:0/1191506616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb40146750 0x7feb401e0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.020+0000 7feb45844640 1 -- 172.21.15.29:0/1191506616 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feb400aa8f0 msgr2=0x7feb4014a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.020+0000 7feb45844640 1 --2- 172.21.15.29:0/1191506616 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feb400aa8f0 0x7feb4014a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.020+0000 7feb44842640 1 --2- 172.21.15.29:0/1191506616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb400ad330 0x7feb40139e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.020+0000 7feb45844640 1 -- 172.21.15.29:0/1191506616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb400ad330 msgr2=0x7feb40139e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.020+0000 7feb45844640 1 --2- 172.21.15.29:0/1191506616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb400ad330 0x7feb40139e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:04.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.020+0000 7feb45844640 1 -- 172.21.15.29:0/1191506616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb4014a520 con 0x7feb40146750 2024-04-25T19:44:04.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.020+0000 7feb45043640 1 --2- 172.21.15.29:0/1191506616 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feb400aa8f0 0x7feb4014a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:04.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.020+0000 7feb45844640 1 --2- 172.21.15.29:0/1191506616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb40146750 0x7feb401e0b50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7feb4014b230 tx=0x7feb3000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:04.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.021+0000 7feb3dffb640 1 -- 172.21.15.29:0/1191506616 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb30003b70 con 0x7feb40146750 2024-04-25T19:44:04.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.021+0000 7feb3dffb640 1 -- 172.21.15.29:0/1191506616 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb30003d10 con 0x7feb40146750 2024-04-25T19:44:04.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.021+0000 7feb472a7640 1 -- 172.21.15.29:0/1191506616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb4013e1f0 con 0x7feb40146750 2024-04-25T19:44:04.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.021+0000 7feb3dffb640 1 -- 172.21.15.29:0/1191506616 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb30005310 con 0x7feb40146750 2024-04-25T19:44:04.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.021+0000 7feb472a7640 1 -- 172.21.15.29:0/1191506616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb40140220 con 0x7feb40146750 2024-04-25T19:44:04.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.022+0000 7feb472a7640 1 -- 172.21.15.29:0/1191506616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb400507b0 con 0x7feb40146750 2024-04-25T19:44:04.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.024+0000 7feb3dffb640 1 -- 172.21.15.29:0/1191506616 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7feb300054b0 con 0x7feb40146750 2024-04-25T19:44:04.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.024+0000 7feb3dffb640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:04.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.025+0000 7feb3dffb640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:04.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.025+0000 7feb3dffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:04.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.025+0000 7feb3dffb640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:04.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.025+0000 7feb3dffb640 1 --2- 172.21.15.29:0/1191506616 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7feb2c0ac530 0x7feb2c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:04.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.025+0000 7feb3dffb640 1 -- 172.21.15.29:0/1191506616 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 3471+0+0 (secure 0 0 0) 0x7feb300d97e0 con 0x7feb40146750 2024-04-25T19:44:04.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.025+0000 7feb45043640 1 --2- 172.21.15.29:0/1191506616 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7feb2c0ac530 0x7feb2c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:04.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.026+0000 7feb45043640 1 --2- 172.21.15.29:0/1191506616 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7feb2c0ac530 0x7feb2c0aea40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7feb380095e0 tx=0x7feb38009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:04.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.029+0000 7feb3dffb640 1 -- 172.21.15.29:0/1191506616 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feb300a47b0 con 0x7feb40146750 2024-04-25T19:44:04.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.212+0000 7feb472a7640 1 -- 172.21.15.29:0/1191506616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) -- 0x7feb400aa6d0 con 0x7feb40146750 2024-04-25T19:44:04.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.891+0000 7feb3dffb640 1 -- 172.21.15.29:0/1191506616 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v15) ==== 173+0+0 (secure 0 0 0) 0x7feb300a40b0 con 0x7feb40146750 2024-04-25T19:44:04.893 INFO:teuthology.orchestra.run.smithi029.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-25T19:44:04.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.895+0000 7feb472a7640 10 mgrc shutdown 2024-04-25T19:44:04.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.895+0000 7feb472a7640 1 -- 172.21.15.29:0/1191506616 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7feb2c0ac530 msgr2=0x7feb2c0aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:04.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.895+0000 7feb472a7640 1 --2- 172.21.15.29:0/1191506616 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7feb2c0ac530 0x7feb2c0aea40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7feb380095e0 tx=0x7feb38009190 comp rx=0 tx=0).stop 2024-04-25T19:44:04.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.895+0000 7feb472a7640 1 -- 172.21.15.29:0/1191506616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb40146750 msgr2=0x7feb401e0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:04.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.895+0000 7feb472a7640 1 --2- 172.21.15.29:0/1191506616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb40146750 0x7feb401e0b50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7feb4014b230 tx=0x7feb3000cb90 comp rx=0 tx=0).stop 2024-04-25T19:44:04.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.895+0000 7feb472a7640 1 -- 172.21.15.29:0/1191506616 shutdown_connections 2024-04-25T19:44:04.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.895+0000 7feb472a7640 1 --2- 172.21.15.29:0/1191506616 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7feb2c0ac530 0x7feb2c0aea40 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:04.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.895+0000 7feb472a7640 1 --2- 172.21.15.29:0/1191506616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb40146750 0x7feb401e0b50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:04.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.895+0000 7feb472a7640 1 --2- 172.21.15.29:0/1191506616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb400ad330 0x7feb40139e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:04.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.895+0000 7feb472a7640 1 --2- 172.21.15.29:0/1191506616 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feb400aa8f0 0x7feb4014a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:04.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.895+0000 7feb472a7640 1 -- 172.21.15.29:0/1191506616 >> 172.21.15.29:0/1191506616 conn(0x7feb400ab2c0 msgr2=0x7feb40146ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:04.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.895+0000 7feb472a7640 1 -- 172.21.15.29:0/1191506616 shutdown_connections 2024-04-25T19:44:04.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:04.895+0000 7feb472a7640 1 -- 172.21.15.29:0/1191506616 wait complete. 2024-04-25T19:44:04.911 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-25T19:44:04.911 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T19:44:04.912 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T19:44:04.914 INFO:tasks.ceph.mds.a:Started 2024-04-25T19:44:04.914 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T19:44:04.914 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T19:44:04.916 INFO:tasks.ceph.mds.c:Started 2024-04-25T19:44:04.916 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T19:44:04.916 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T19:44:04.918 INFO:tasks.ceph.mds.b:Started 2024-04-25T19:44:04.919 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-25T19:44:04.919 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:44:05.008 INFO:tasks.ceph.mds.b.smithi102.stdout:starting mds.b at 2024-04-25T19:44:05.009 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2024-04-25T19:44:05.010 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2024-04-25T19:44:05.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.041+0000 7ff81e3bb640 1 Processor -- start 2024-04-25T19:44:05.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.041+0000 7ff81e3bb640 1 -- start start 2024-04-25T19:44:05.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.042+0000 7ff81e3bb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8180aa120 0x7ff8180aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:05.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.042+0000 7ff81e3bb640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff8180ac4e0 0x7ff8180ac8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:05.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.042+0000 7ff81e3bb640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff81806dce0 con 0x7ff81814e740 2024-04-25T19:44:05.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.042+0000 7ff81e3bb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7ff81814f3d0 con 0x7ff8180aa120 2024-04-25T19:44:05.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.042+0000 7ff817fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8180aa120 0x7ff8180aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:05.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.042+0000 7ff81e3bb640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7ff8180b81d0 con 0x7ff8180ac4e0 2024-04-25T19:44:05.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.042+0000 7ff817fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8180aa120 0x7ff8180aa530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49676/0 (socket says 172.21.15.29:49676) 2024-04-25T19:44:05.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.042+0000 7ff817fff640 1 -- 172.21.15.29:0/2198010255 learned_addr learned my addr 172.21.15.29:0/2198010255 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:05.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.042+0000 7ff8177fe640 1 --2- 172.21.15.29:0/2198010255 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff8180ac4e0 0x7ff8180ac8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:05.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.043+0000 7ff817fff640 1 -- 172.21.15.29:0/2198010255 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff8180ac4e0 msgr2=0x7ff8180ac8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:05.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.043+0000 7ff817fff640 1 --2- 172.21.15.29:0/2198010255 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff8180ac4e0 0x7ff8180ac8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.043+0000 7ff817fff640 1 -- 172.21.15.29:0/2198010255 >> v1:172.21.15.102:6789/0 conn(0x7ff81814e740 legacy=0x7ff81814eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:05.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.043+0000 7ff817fff640 1 -- 172.21.15.29:0/2198010255 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff818141890 con 0x7ff8180aa120 2024-04-25T19:44:05.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.043+0000 7ff8177fe640 1 --2- 172.21.15.29:0/2198010255 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff8180ac4e0 0x7ff8180ac8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:05.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.044+0000 7ff817fff640 1 --2- 172.21.15.29:0/2198010255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8180aa120 0x7ff8180aa530 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff808009870 tx=0x7ff80802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f571542470d4cdf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:05.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.044+0000 7ff816ffd640 1 -- 172.21.15.29:0/2198010255 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80802f840 con 0x7ff8180aa120 2024-04-25T19:44:05.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.044+0000 7ff816ffd640 1 -- 172.21.15.29:0/2198010255 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff80802f9e0 con 0x7ff8180aa120 2024-04-25T19:44:05.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.044+0000 7ff816ffd640 1 -- 172.21.15.29:0/2198010255 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80802fdb0 con 0x7ff8180aa120 2024-04-25T19:44:05.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.044+0000 7ff81e3bb640 1 -- 172.21.15.29:0/2198010255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8180aa120 msgr2=0x7ff8180aa530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.044+0000 7ff81e3bb640 1 --2- 172.21.15.29:0/2198010255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8180aa120 0x7ff8180aa530 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff808009870 tx=0x7ff80802ba20 comp rx=0 tx=0).stop 2024-04-25T19:44:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.044+0000 7ff81e3bb640 1 -- 172.21.15.29:0/2198010255 shutdown_connections 2024-04-25T19:44:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.044+0000 7ff81e3bb640 1 --2- 172.21.15.29:0/2198010255 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff8180ac4e0 0x7ff8180ac8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.044+0000 7ff81e3bb640 1 --2- 172.21.15.29:0/2198010255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8180aa120 0x7ff8180aa530 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.044+0000 7ff81e3bb640 1 -- 172.21.15.29:0/2198010255 >> 172.21.15.29:0/2198010255 conn(0x7ff8180bc3b0 msgr2=0x7ff8180bc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.044+0000 7ff81e3bb640 1 -- 172.21.15.29:0/2198010255 shutdown_connections 2024-04-25T19:44:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.044+0000 7ff81e3bb640 1 -- 172.21.15.29:0/2198010255 wait complete. 2024-04-25T19:44:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.046+0000 7ff81e3bb640 1 Processor -- start 2024-04-25T19:44:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.046+0000 7ff81e3bb640 1 -- start start 2024-04-25T19:44:05.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.047+0000 7ff81e3bb640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff8180aa120 0x7ff8180bae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.047+0000 7ff81e3bb640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8180ac4e0 0x7ff8180a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.047+0000 7ff81e3bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff81814e740 0x7ff818148690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.047+0000 7ff81e3bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff818155c50 con 0x7ff81814e740 2024-04-25T19:44:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.047+0000 7ff81e3bb640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff818155ad0 con 0x7ff8180ac4e0 2024-04-25T19:44:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.047+0000 7ff81e3bb640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7ff818155dd0 con 0x7ff8180aa120 2024-04-25T19:44:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.047+0000 7ff81c958640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff81814e740 0x7ff818148690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.047+0000 7ff817fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff8180aa120 0x7ff8180bae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.047+0000 7ff81c958640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff81814e740 0x7ff818148690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49682/0 (socket says 172.21.15.29:49682) 2024-04-25T19:44:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.047+0000 7ff81c958640 1 -- 172.21.15.29:0/2923173906 learned_addr learned my addr 172.21.15.29:0/2923173906 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.047+0000 7ff817fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff8180aa120 0x7ff8180bae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:60276/0 (socket says 172.21.15.29:60276) 2024-04-25T19:44:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.047+0000 7ff8177fe640 1 --2- 172.21.15.29:0/2923173906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8180ac4e0 0x7ff8180a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.048+0000 7ff81c958640 1 -- 172.21.15.29:0/2923173906 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff8180aa120 msgr2=0x7ff8180bae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.048+0000 7ff81c958640 1 --2- 172.21.15.29:0/2923173906 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff8180aa120 0x7ff8180bae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.048+0000 7ff81c958640 1 -- 172.21.15.29:0/2923173906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8180ac4e0 msgr2=0x7ff8180a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:05.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.048+0000 7ff81c958640 1 --2- 172.21.15.29:0/2923173906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8180ac4e0 0x7ff8180a1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.048+0000 7ff81c958640 1 -- 172.21.15.29:0/2923173906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8180bbd40 con 0x7ff81814e740 2024-04-25T19:44:05.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.048+0000 7ff817fff640 1 --2- 172.21.15.29:0/2923173906 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff8180aa120 0x7ff8180bae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:05.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.048+0000 7ff81c958640 1 --2- 172.21.15.29:0/2923173906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff81814e740 0x7ff818148690 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff81000c760 tx=0x7ff81000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:05.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.048+0000 7ff814ff9640 1 -- 172.21.15.29:0/2923173906 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff810003b70 con 0x7ff81814e740 2024-04-25T19:44:05.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.048+0000 7ff814ff9640 1 -- 172.21.15.29:0/2923173906 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff810003d10 con 0x7ff81814e740 2024-04-25T19:44:05.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.048+0000 7ff81e3bb640 1 -- 172.21.15.29:0/2923173906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8180bba60 con 0x7ff81814e740 2024-04-25T19:44:05.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.048+0000 7ff814ff9640 1 -- 172.21.15.29:0/2923173906 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81000aa90 con 0x7ff81814e740 2024-04-25T19:44:05.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.048+0000 7ff81e3bb640 1 -- 172.21.15.29:0/2923173906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8180bb390 con 0x7ff81814e740 2024-04-25T19:44:05.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.049+0000 7ff81e3bb640 1 -- 172.21.15.29:0/2923173906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8180a22d0 con 0x7ff81814e740 2024-04-25T19:44:05.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.050+0000 7ff814ff9640 1 -- 172.21.15.29:0/2923173906 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7ff81000ac30 con 0x7ff81814e740 2024-04-25T19:44:05.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.050+0000 7ff814ff9640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:05.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.051+0000 7ff814ff9640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:05.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.051+0000 7ff814ff9640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:05.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.051+0000 7ff814ff9640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:05.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.051+0000 7ff814ff9640 1 --2- 172.21.15.29:0/2923173906 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7ff8040ac530 0x7ff8040aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:05.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.051+0000 7ff814ff9640 1 -- 172.21.15.29:0/2923173906 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3482+0+0 (secure 0 0 0) 0x7ff8100d8dc0 con 0x7ff81814e740 2024-04-25T19:44:05.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.051+0000 7ff817fff640 1 --2- 172.21.15.29:0/2923173906 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7ff8040ac530 0x7ff8040aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:05.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.052+0000 7ff817fff640 1 --2- 172.21.15.29:0/2923173906 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7ff8040ac530 0x7ff8040aea40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff808036000 tx=0x7ff808008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:05.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.055+0000 7ff814ff9640 1 -- 172.21.15.29:0/2923173906 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff8100a3d80 con 0x7ff81814e740 2024-04-25T19:44:05.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.244+0000 7ff81e3bb640 1 -- 172.21.15.29:0/2923173906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff8180507b0 con 0x7ff81814e740 2024-04-25T19:44:05.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.245+0000 7ff814ff9640 1 -- 172.21.15.29:0/2923173906 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+9747 (secure 0 0 0) 0x7ff8100a3680 con 0x7ff81814e740 2024-04-25T19:44:05.246 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:05.246 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":15,"fsid":"1b3c39b5-e787-409e-b0bd-f4f491c89d72","created":"2024-04-25T19:43:47.854697+0000","modified":"2024-04-25T19:44:04.887002+0000","last_up_change":"2024-04-25T19:43:57.853237+0000","last_in_change":"2024-04-25T19:43:49.022776+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:43:58.096700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T19:44:03.617011+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d805223-ee84-494f-bd31-59091906aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6801","nonce":695704147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6803","nonce":695704147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6807","nonce":695704147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6805","nonce":695704147}]},"public_addr":"172.21.15.29:6801/695704147","cluster_addr":"172.21.15.29:6803/695704147","heartbeat_back_addr":"172.21.15.29:6807/695704147","heartbeat_front_addr":"172.21.15.29:6805/695704147","state":["exists","up"]},{"osd":1,"uuid":"607c5038-38af-440f-8371-e71758549920","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6814","nonce":3135821452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6817","nonce":3135821452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6823","nonce":3135821452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6821","nonce":3135821452}]},"public_addr":"172.21.15.102:6814/3135821452","cluster_addr":"172.21.15.102:6817/3135821452","heartbeat_back_addr":"172.21.15.102:6823/3135821452","heartbeat_front_addr":"172.21.15.102:6821/3135821452","state":["exists","up"]},{"osd":2,"uuid":"6859ae76-54bb-4c55-b0dc-5e2735c0d31c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6801","nonce":2071844144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6803","nonce":2071844144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6811","nonce":2071844144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6806","nonce":2071844144}]},"public_addr":"172.21.15.102:6801/2071844144","cluster_addr":"172.21.15.102:6803/2071844144","heartbeat_back_addr":"172.21.15.102:6811/2071844144","heartbeat_front_addr":"172.21.15.102:6806/2071844144","state":["exists","up"]},{"osd":3,"uuid":"a98b7b88-5206-4f22-84a6-78212e0d5f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6805","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6807","nonce":339944984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6810","nonce":339944984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6820","nonce":339944984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6815","nonce":339944984}]},"public_addr":"172.21.15.102:6807/339944984","cluster_addr":"172.21.15.102:6810/339944984","heartbeat_back_addr":"172.21.15.102:6820/339944984","heartbeat_front_addr":"172.21.15.102:6815/339944984","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.121529+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.201155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.264387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.198713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:44:05.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.247+0000 7ff81e3bb640 10 mgrc shutdown 2024-04-25T19:44:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.247+0000 7ff81e3bb640 1 -- 172.21.15.29:0/2923173906 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7ff8040ac530 msgr2=0x7ff8040aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.247+0000 7ff81e3bb640 1 --2- 172.21.15.29:0/2923173906 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7ff8040ac530 0x7ff8040aea40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff808036000 tx=0x7ff808008030 comp rx=0 tx=0).stop 2024-04-25T19:44:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.247+0000 7ff81e3bb640 1 -- 172.21.15.29:0/2923173906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff81814e740 msgr2=0x7ff818148690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.247+0000 7ff81e3bb640 1 --2- 172.21.15.29:0/2923173906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff81814e740 0x7ff818148690 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff81000c760 tx=0x7ff81000cc20 comp rx=0 tx=0).stop 2024-04-25T19:44:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.247+0000 7ff81e3bb640 1 -- 172.21.15.29:0/2923173906 shutdown_connections 2024-04-25T19:44:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.247+0000 7ff81e3bb640 1 --2- 172.21.15.29:0/2923173906 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7ff8040ac530 0x7ff8040aea40 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.247+0000 7ff81e3bb640 1 --2- 172.21.15.29:0/2923173906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff81814e740 0x7ff818148690 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.247+0000 7ff81e3bb640 1 --2- 172.21.15.29:0/2923173906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8180ac4e0 0x7ff8180a1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.247+0000 7ff81e3bb640 1 --2- 172.21.15.29:0/2923173906 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff8180aa120 0x7ff8180bae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.247+0000 7ff81e3bb640 1 -- 172.21.15.29:0/2923173906 >> 172.21.15.29:0/2923173906 conn(0x7ff8180bc3b0 msgr2=0x7ff8180ad3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.247+0000 7ff81e3bb640 1 -- 172.21.15.29:0/2923173906 shutdown_connections 2024-04-25T19:44:05.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.247+0000 7ff81e3bb640 1 -- 172.21.15.29:0/2923173906 wait complete. 2024-04-25T19:44:05.259 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:43:58.096700+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '13', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4, 'score_stable': 4, 'optimal_score': 0.5, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-25T19:44:03.617011+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T19:44:05.260 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:44:05.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.371+0000 7f7364754640 1 Processor -- start 2024-04-25T19:44:05.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.371+0000 7f7364754640 1 -- start start 2024-04-25T19:44:05.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.372+0000 7f7364754640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f735c146640 0x7f735c0aa840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:05.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.372+0000 7f7364754640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f735c14f3f0 0x7f735c151810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:05.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.372+0000 7f7364754640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f735c0ad500 con 0x7f735c06df40 2024-04-25T19:44:05.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.372+0000 7f7364754640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f735c0ad7c0 con 0x7f735c146640 2024-04-25T19:44:05.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.372+0000 7f7364754640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f735c0aad80 con 0x7f735c14f3f0 2024-04-25T19:44:05.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.372+0000 7f73624f0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f735c146640 0x7f735c0aa840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:05.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.372+0000 7f7361cef640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f735c06df40 0x7f735c0aca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55414/0 (socket says 172.21.15.29:55414) 2024-04-25T19:44:05.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.372+0000 7f7361cef640 1 -- 172.21.15.29:0/2757213634 learned_addr learned my addr 172.21.15.29:0/2757213634 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:05.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.372+0000 7f7362cf1640 1 --2- 172.21.15.29:0/2757213634 >> v2:172.21.15.102:3300/0 conn(0x7f735c14f3f0 0x7f735c151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:05.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.373+0000 7f73624f0640 1 -- 172.21.15.29:0/2757213634 >> v2:172.21.15.102:3300/0 conn(0x7f735c14f3f0 msgr2=0x7f735c151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:05.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.373+0000 7f73624f0640 1 --2- 172.21.15.29:0/2757213634 >> v2:172.21.15.102:3300/0 conn(0x7f735c14f3f0 0x7f735c151810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.373+0000 7f73624f0640 1 -- 172.21.15.29:0/2757213634 >> v1:172.21.15.29:6789/0 conn(0x7f735c06df40 legacy=0x7f735c0aca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:05.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.373+0000 7f73614ee640 1 -- 172.21.15.29:0/2757213634 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1522854612 0 0) 0x7f735c0ad500 con 0x7f735c06df40 2024-04-25T19:44:05.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.373+0000 7f73624f0640 1 -- 172.21.15.29:0/2757213634 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f735c0ae650 con 0x7f735c146640 2024-04-25T19:44:05.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.373+0000 7f73624f0640 1 --2- 172.21.15.29:0/2757213634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f735c146640 0x7f735c0aa840 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f735c0aa390 tx=0x7f735402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a0851c9885c247 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:05.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.373+0000 7f73614ee640 1 -- 172.21.15.29:0/2757213634 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7354034070 con 0x7f735c146640 2024-04-25T19:44:05.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.374+0000 7f73614ee640 1 -- 172.21.15.29:0/2757213634 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7354030610 con 0x7f735c146640 2024-04-25T19:44:05.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.374+0000 7f73614ee640 1 -- 172.21.15.29:0/2757213634 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73540309e0 con 0x7f735c146640 2024-04-25T19:44:05.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.374+0000 7f7364754640 1 -- 172.21.15.29:0/2757213634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f735c146640 msgr2=0x7f735c0aa840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:05.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.374+0000 7f7364754640 1 --2- 172.21.15.29:0/2757213634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f735c146640 0x7f735c0aa840 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f735c0aa390 tx=0x7f735402b960 comp rx=0 tx=0).stop 2024-04-25T19:44:05.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.374+0000 7f7364754640 1 -- 172.21.15.29:0/2757213634 shutdown_connections 2024-04-25T19:44:05.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.374+0000 7f7364754640 1 --2- 172.21.15.29:0/2757213634 >> v2:172.21.15.102:3300/0 conn(0x7f735c14f3f0 0x7f735c151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.374+0000 7f7364754640 1 --2- 172.21.15.29:0/2757213634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f735c146640 0x7f735c0aa840 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.374+0000 7f7364754640 1 -- 172.21.15.29:0/2757213634 >> 172.21.15.29:0/2757213634 conn(0x7f735c0ab710 msgr2=0x7f735c0bbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:05.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.374+0000 7f7364754640 1 -- 172.21.15.29:0/2757213634 shutdown_connections 2024-04-25T19:44:05.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.374+0000 7f7364754640 1 -- 172.21.15.29:0/2757213634 wait complete. 2024-04-25T19:44:05.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.376+0000 7f7364754640 1 Processor -- start 2024-04-25T19:44:05.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.376+0000 7f7364754640 1 -- start start 2024-04-25T19:44:05.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.377+0000 7f7364754640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f735c06df40 0x7f735c14aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:05.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.377+0000 7f7364754640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f735c146640 0x7f735c14d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:05.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.377+0000 7f7364754640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f735c14f3f0 0x7f735c14be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:05.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.377+0000 7f7364754640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f735c06e4d0 con 0x7f735c14f3f0 2024-04-25T19:44:05.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.377+0000 7f7364754640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f735c06e350 con 0x7f735c146640 2024-04-25T19:44:05.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.377+0000 7f7364754640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f735c151d50 con 0x7f735c06df40 2024-04-25T19:44:05.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.377+0000 7f73624f0640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f735c06df40 0x7f735c14aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:05.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.377+0000 7f7362cf1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f735c14f3f0 0x7f735c14be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:05.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.377+0000 7f73624f0640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f735c06df40 0x7f735c14aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:60286/0 (socket says 172.21.15.29:60286) 2024-04-25T19:44:05.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.377+0000 7f7362cf1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f735c14f3f0 0x7f735c14be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49692/0 (socket says 172.21.15.29:49692) 2024-04-25T19:44:05.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.377+0000 7f73624f0640 1 -- 172.21.15.29:0/2125732155 learned_addr learned my addr 172.21.15.29:0/2125732155 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:05.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.377+0000 7f7361cef640 1 --2- 172.21.15.29:0/2125732155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f735c146640 0x7f735c14d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:05.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.377+0000 7f73624f0640 1 -- 172.21.15.29:0/2125732155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f735c146640 msgr2=0x7f735c14d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:05.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.377+0000 7f73624f0640 1 --2- 172.21.15.29:0/2125732155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f735c146640 0x7f735c14d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.377+0000 7f73624f0640 1 -- 172.21.15.29:0/2125732155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f735c14f3f0 msgr2=0x7f735c14be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:05.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.377+0000 7f73624f0640 1 --2- 172.21.15.29:0/2125732155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f735c14f3f0 0x7f735c14be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.377+0000 7f73624f0640 1 -- 172.21.15.29:0/2125732155 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f735c14c8a0 con 0x7f735c06df40 2024-04-25T19:44:05.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.377+0000 7f7362cf1640 1 --2- 172.21.15.29:0/2125732155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f735c14f3f0 0x7f735c14be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:05.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.377+0000 7f73624f0640 1 --2- 172.21.15.29:0/2125732155 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f735c06df40 0x7f735c14aa50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7354009890 tx=0x7f73540046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:05.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.378+0000 7f7352ffd640 1 -- 172.21.15.29:0/2125732155 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7354034040 con 0x7f735c06df40 2024-04-25T19:44:05.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.378+0000 7f7352ffd640 1 -- 172.21.15.29:0/2125732155 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f735403aaa0 con 0x7f735c06df40 2024-04-25T19:44:05.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.378+0000 7f7364754640 1 -- 172.21.15.29:0/2125732155 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f735c14b960 con 0x7f735c06df40 2024-04-25T19:44:05.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.378+0000 7f7352ffd640 1 -- 172.21.15.29:0/2125732155 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73540443f0 con 0x7f735c06df40 2024-04-25T19:44:05.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.378+0000 7f7364754640 1 -- 172.21.15.29:0/2125732155 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f735c14c380 con 0x7f735c06df40 2024-04-25T19:44:05.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.379+0000 7f7364754640 1 -- 172.21.15.29:0/2125732155 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f735c0507b0 con 0x7f735c06df40 2024-04-25T19:44:05.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.380+0000 7f7352ffd640 1 -- 172.21.15.29:0/2125732155 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f7354037070 con 0x7f735c06df40 2024-04-25T19:44:05.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.380+0000 7f7352ffd640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:05.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.381+0000 7f7352ffd640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:05.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.381+0000 7f7352ffd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:05.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.381+0000 7f7352ffd640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:05.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.381+0000 7f7352ffd640 1 --2- 172.21.15.29:0/2125732155 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f73400ac4e0 0x7f73400ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:05.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.381+0000 7f7352ffd640 1 -- 172.21.15.29:0/2125732155 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3482+0+0 (secure 0 0 0) 0x7f73540fb6f0 con 0x7f735c06df40 2024-04-25T19:44:05.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.381+0000 7f7361cef640 1 --2- 172.21.15.29:0/2125732155 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f73400ac4e0 0x7f73400ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:05.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.382+0000 7f7361cef640 1 --2- 172.21.15.29:0/2125732155 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f73400ac4e0 0x7f73400ae9f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f734c007fd0 tx=0x7f734c007de0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:05.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.384+0000 7f7352ffd640 1 -- 172.21.15.29:0/2125732155 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7354052070 con 0x7f735c06df40 2024-04-25T19:44:05.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.565+0000 7f7364754640 1 -- 172.21.15.29:0/2125732155 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f735c0ba890 con 0x7f735c06df40 2024-04-25T19:44:05.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.566+0000 7f7352ffd640 1 -- 172.21.15.29:0/2125732155 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v15) ==== 93+0+10 (secure 0 0 0) 0x7f735403f070 con 0x7f735c06df40 2024-04-25T19:44:05.567 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-04-25T19:44:05.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.568+0000 7f7364754640 10 mgrc shutdown 2024-04-25T19:44:05.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.568+0000 7f7364754640 1 -- 172.21.15.29:0/2125732155 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f73400ac4e0 msgr2=0x7f73400ae9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:05.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.568+0000 7f7364754640 1 --2- 172.21.15.29:0/2125732155 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f73400ac4e0 0x7f73400ae9f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f734c007fd0 tx=0x7f734c007de0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.568+0000 7f7364754640 1 -- 172.21.15.29:0/2125732155 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f735c06df40 msgr2=0x7f735c14aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.568+0000 7f7364754640 1 --2- 172.21.15.29:0/2125732155 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f735c06df40 0x7f735c14aa50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7354009890 tx=0x7f73540046c0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.568+0000 7f7364754640 1 -- 172.21.15.29:0/2125732155 shutdown_connections 2024-04-25T19:44:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.568+0000 7f7364754640 1 --2- 172.21.15.29:0/2125732155 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f73400ac4e0 0x7f73400ae9f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.568+0000 7f7364754640 1 --2- 172.21.15.29:0/2125732155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f735c14f3f0 0x7f735c14be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.568+0000 7f7364754640 1 --2- 172.21.15.29:0/2125732155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f735c146640 0x7f735c14d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.568+0000 7f7364754640 1 --2- 172.21.15.29:0/2125732155 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f735c06df40 0x7f735c14aa50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.568+0000 7f7364754640 1 -- 172.21.15.29:0/2125732155 >> 172.21.15.29:0/2125732155 conn(0x7f735c0ab710 msgr2=0x7f735c0684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.568+0000 7f7364754640 1 -- 172.21.15.29:0/2125732155 shutdown_connections 2024-04-25T19:44:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.568+0000 7f7364754640 1 -- 172.21.15.29:0/2125732155 wait complete. 2024-04-25T19:44:05.580 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-04-25T19:44:05.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.692+0000 7f3892dbd640 1 Processor -- start 2024-04-25T19:44:05.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.692+0000 7f3892dbd640 1 -- start start 2024-04-25T19:44:05.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.693+0000 7f3892dbd640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f388c06da90 0x7f388c0a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:05.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.693+0000 7f3892dbd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f388c1472d0 0x7f388c1476e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:05.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.693+0000 7f3892dbd640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f388c146c60 con 0x7f388c0acd40 2024-04-25T19:44:05.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.693+0000 7f3892dbd640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f388c0a5c10 con 0x7f388c06da90 2024-04-25T19:44:05.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.693+0000 7f3892dbd640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f388c0ad9d0 con 0x7f388c1472d0 2024-04-25T19:44:05.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.693+0000 7f3883fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f388c1472d0 0x7f388c1476e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:05.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.693+0000 7f3883fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f388c1472d0 0x7f388c1476e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:60296/0 (socket says 172.21.15.29:60296) 2024-04-25T19:44:05.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.693+0000 7f3883fff640 1 -- 172.21.15.29:0/1593798907 learned_addr learned my addr 172.21.15.29:0/1593798907 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:05.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.693+0000 7f3890b59640 1 --2- 172.21.15.29:0/1593798907 >> v2:172.21.15.102:3300/0 conn(0x7f388c06da90 0x7f388c0a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:05.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.693+0000 7f3883fff640 1 -- 172.21.15.29:0/1593798907 >> v2:172.21.15.102:3300/0 conn(0x7f388c06da90 msgr2=0x7f388c0a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:05.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.693+0000 7f3883fff640 1 --2- 172.21.15.29:0/1593798907 >> v2:172.21.15.102:3300/0 conn(0x7f388c06da90 0x7f388c0a56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.693+0000 7f3883fff640 1 -- 172.21.15.29:0/1593798907 >> v1:172.21.15.102:6789/0 conn(0x7f388c0acd40 legacy=0x7f388c0ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:05.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.693+0000 7f3883fff640 1 -- 172.21.15.29:0/1593798907 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f388c152400 con 0x7f388c1472d0 2024-04-25T19:44:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.694+0000 7f3883fff640 1 --2- 172.21.15.29:0/1593798907 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f388c1472d0 0x7f388c1476e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f387c009870 tx=0x7f387c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bc44545b6309f6f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.694+0000 7f38837fe640 1 -- 172.21.15.29:0/1593798907 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c034070 con 0x7f388c1472d0 2024-04-25T19:44:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.694+0000 7f38837fe640 1 -- 172.21.15.29:0/1593798907 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f387c030840 con 0x7f388c1472d0 2024-04-25T19:44:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.694+0000 7f38837fe640 1 -- 172.21.15.29:0/1593798907 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c030c10 con 0x7f388c1472d0 2024-04-25T19:44:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.695+0000 7f3892dbd640 1 -- 172.21.15.29:0/1593798907 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f388c1472d0 msgr2=0x7f388c1476e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.695+0000 7f3892dbd640 1 --2- 172.21.15.29:0/1593798907 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f388c1472d0 0x7f388c1476e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f387c009870 tx=0x7f387c02b980 comp rx=0 tx=0).stop 2024-04-25T19:44:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.695+0000 7f3892dbd640 1 -- 172.21.15.29:0/1593798907 shutdown_connections 2024-04-25T19:44:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.695+0000 7f3892dbd640 1 --2- 172.21.15.29:0/1593798907 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f388c1472d0 0x7f388c1476e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.695+0000 7f3892dbd640 1 --2- 172.21.15.29:0/1593798907 >> v2:172.21.15.102:3300/0 conn(0x7f388c06da90 0x7f388c0a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.695+0000 7f3892dbd640 1 -- 172.21.15.29:0/1593798907 >> 172.21.15.29:0/1593798907 conn(0x7f388c0bc3b0 msgr2=0x7f388c0bc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.695+0000 7f3892dbd640 1 -- 172.21.15.29:0/1593798907 shutdown_connections 2024-04-25T19:44:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.695+0000 7f3892dbd640 1 -- 172.21.15.29:0/1593798907 wait complete. 2024-04-25T19:44:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.696+0000 7f3892dbd640 1 Processor -- start 2024-04-25T19:44:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.697+0000 7f3892dbd640 1 -- start start 2024-04-25T19:44:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.698+0000 7f3892dbd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f388c06da90 0x7f388c0a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.698+0000 7f3892dbd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f388c0acd40 0x7f388c0a30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.698+0000 7f3892dbd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f388c1472d0 0x7f388c0b8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.698+0000 7f3892dbd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f388c1517c0 con 0x7f388c1472d0 2024-04-25T19:44:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.698+0000 7f3892dbd640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f388c151640 con 0x7f388c0acd40 2024-04-25T19:44:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.698+0000 7f3892dbd640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f388c151940 con 0x7f388c06da90 2024-04-25T19:44:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.698+0000 7f3890b59640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f388c06da90 0x7f388c0a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.698+0000 7f3890b59640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f388c06da90 0x7f388c0a0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:60312/0 (socket says 172.21.15.29:60312) 2024-04-25T19:44:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.698+0000 7f389135a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f388c1472d0 0x7f388c0b8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.698+0000 7f3890b59640 1 -- 172.21.15.29:0/652291642 learned_addr learned my addr 172.21.15.29:0/652291642 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.698+0000 7f3883fff640 1 --2- 172.21.15.29:0/652291642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f388c0acd40 0x7f388c0a30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.698+0000 7f3890b59640 1 -- 172.21.15.29:0/652291642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f388c0acd40 msgr2=0x7f388c0a30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.698+0000 7f3890b59640 1 --2- 172.21.15.29:0/652291642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f388c0acd40 0x7f388c0a30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.698+0000 7f3890b59640 1 -- 172.21.15.29:0/652291642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f388c1472d0 msgr2=0x7f388c0b8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.698+0000 7f3890b59640 1 --2- 172.21.15.29:0/652291642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f388c1472d0 0x7f388c0b8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.698+0000 7f3890b59640 1 -- 172.21.15.29:0/652291642 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f388c0a6990 con 0x7f388c06da90 2024-04-25T19:44:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.698+0000 7f389135a640 1 --2- 172.21.15.29:0/652291642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f388c1472d0 0x7f388c0b8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.698+0000 7f3890b59640 1 --2- 172.21.15.29:0/652291642 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f388c06da90 0x7f388c0a0b60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f388800e8a0 tx=0x7f388800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.698+0000 7f38817fa640 1 -- 172.21.15.29:0/652291642 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3888017070 con 0x7f388c06da90 2024-04-25T19:44:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.699+0000 7f38817fa640 1 -- 172.21.15.29:0/652291642 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f388800c940 con 0x7f388c06da90 2024-04-25T19:44:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.699+0000 7f3892dbd640 1 -- 172.21.15.29:0/652291642 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f388c0a6750 con 0x7f388c06da90 2024-04-25T19:44:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.699+0000 7f38817fa640 1 -- 172.21.15.29:0/652291642 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f388800cd30 con 0x7f388c06da90 2024-04-25T19:44:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.699+0000 7f3892dbd640 1 -- 172.21.15.29:0/652291642 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f388c0a7080 con 0x7f388c06da90 2024-04-25T19:44:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.699+0000 7f3892dbd640 1 -- 172.21.15.29:0/652291642 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f388c0a7ca0 con 0x7f388c06da90 2024-04-25T19:44:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.702+0000 7f38817fa640 1 -- 172.21.15.29:0/652291642 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f388801a070 con 0x7f388c06da90 2024-04-25T19:44:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.702+0000 7f38817fa640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.702+0000 7f38817fa640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.702+0000 7f38817fa640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.702+0000 7f38817fa640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.702+0000 7f38817fa640 1 --2- 172.21.15.29:0/652291642 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f38780ac530 0x7f38780aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.702+0000 7f38817fa640 1 -- 172.21.15.29:0/652291642 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3482+0+0 (secure 0 0 0) 0x7f38880d9560 con 0x7f388c06da90 2024-04-25T19:44:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.702+0000 7f3883fff640 1 --2- 172.21.15.29:0/652291642 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f38780ac530 0x7f38780aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.703+0000 7f3883fff640 1 --2- 172.21.15.29:0/652291642 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f38780ac530 0x7f38780aea40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f387c036000 tx=0x7f387c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.706+0000 7f38817fa640 1 -- 172.21.15.29:0/652291642 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f388c0a7ca0 con 0x7f388c06da90 2024-04-25T19:44:05.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.888+0000 7f3892dbd640 1 -- 172.21.15.29:0/652291642 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7f388c0507b0 con 0x7f388c06da90 2024-04-25T19:44:05.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.889+0000 7f38817fa640 1 -- 172.21.15.29:0/652291642 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v15) ==== 92+0+10 (secure 0 0 0) 0x7f38880a3ed0 con 0x7f388c06da90 2024-04-25T19:44:05.890 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 8 2024-04-25T19:44:05.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.891+0000 7f3892dbd640 10 mgrc shutdown 2024-04-25T19:44:05.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.891+0000 7f3892dbd640 1 -- 172.21.15.29:0/652291642 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f38780ac530 msgr2=0x7f38780aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:05.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.891+0000 7f3892dbd640 1 --2- 172.21.15.29:0/652291642 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f38780ac530 0x7f38780aea40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f387c036000 tx=0x7f387c008030 comp rx=0 tx=0).stop 2024-04-25T19:44:05.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.891+0000 7f3892dbd640 1 -- 172.21.15.29:0/652291642 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f388c06da90 msgr2=0x7f388c0a0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:05.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.891+0000 7f3892dbd640 1 --2- 172.21.15.29:0/652291642 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f388c06da90 0x7f388c0a0b60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f388800e8a0 tx=0x7f388800ed60 comp rx=0 tx=0).stop 2024-04-25T19:44:05.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.891+0000 7f3892dbd640 1 -- 172.21.15.29:0/652291642 shutdown_connections 2024-04-25T19:44:05.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.891+0000 7f3892dbd640 1 --2- 172.21.15.29:0/652291642 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f38780ac530 0x7f38780aea40 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.891+0000 7f3892dbd640 1 --2- 172.21.15.29:0/652291642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f388c1472d0 0x7f388c0b8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.891+0000 7f3892dbd640 1 --2- 172.21.15.29:0/652291642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f388c0acd40 0x7f388c0a30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.891+0000 7f3892dbd640 1 --2- 172.21.15.29:0/652291642 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f388c06da90 0x7f388c0a0b60 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:05.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.891+0000 7f3892dbd640 1 -- 172.21.15.29:0/652291642 >> 172.21.15.29:0/652291642 conn(0x7f388c0bc3b0 msgr2=0x7f388c0a9630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:05.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.891+0000 7f3892dbd640 1 -- 172.21.15.29:0/652291642 shutdown_connections 2024-04-25T19:44:05.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:05.891+0000 7f3892dbd640 1 -- 172.21.15.29:0/652291642 wait complete. 2024-04-25T19:44:05.907 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:44:06.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.017+0000 7f5b5a522640 1 Processor -- start 2024-04-25T19:44:06.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.017+0000 7f5b5a522640 1 -- start start 2024-04-25T19:44:06.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.018+0000 7f5b5a522640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b540731e0 0x7f5b54152550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:06.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.018+0000 7f5b5a522640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b54146bd0 con 0x7f5b54072dd0 2024-04-25T19:44:06.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.018+0000 7f5b5a522640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b54146e60 con 0x7f5b540aae50 2024-04-25T19:44:06.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.018+0000 7f5b5a522640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f5b540744b0 con 0x7f5b540731e0 2024-04-25T19:44:06.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.018+0000 7f5b53fff640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5b540aae50 0x7f5b5406c690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.29:55046/0 (socket says 172.21.15.29:55046) 2024-04-25T19:44:06.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.018+0000 7f5b53fff640 1 -- 172.21.15.29:0/1762047161 learned_addr learned my addr 172.21.15.29:0/1762047161 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:06.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.018+0000 7f5b58abf640 1 --2- 172.21.15.29:0/1762047161 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b540731e0 0x7f5b54152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:06.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.018+0000 7f5b537fe640 1 --1- 172.21.15.29:0/1762047161 >> v1:172.21.15.29:6789/0 conn(0x7f5b54072dd0 0x7f5b541464c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55424/0 (socket says 172.21.15.29:55424) 2024-04-25T19:44:06.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.018+0000 7f5b52ffd640 1 -- 172.21.15.29:0/1762047161 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2522360574 0 0) 0x7f5b54146e60 con 0x7f5b540aae50 2024-04-25T19:44:06.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.018+0000 7f5b52ffd640 1 -- 172.21.15.29:0/1762047161 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b3c003520 con 0x7f5b540aae50 2024-04-25T19:44:06.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.019+0000 7f5b58abf640 1 -- 172.21.15.29:0/1762047161 >> v1:172.21.15.102:6789/0 conn(0x7f5b540aae50 legacy=0x7f5b5406c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:06.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.019+0000 7f5b52ffd640 1 -- 172.21.15.29:0/1762047161 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1162557122 0 0) 0x7f5b54146bd0 con 0x7f5b54072dd0 2024-04-25T19:44:06.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.019+0000 7f5b58abf640 1 -- 172.21.15.29:0/1762047161 >> v1:172.21.15.29:6789/0 conn(0x7f5b54072dd0 legacy=0x7f5b541464c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:06.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.019+0000 7f5b58abf640 1 -- 172.21.15.29:0/1762047161 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b540ab260 con 0x7f5b540731e0 2024-04-25T19:44:06.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.019+0000 7f5b58abf640 1 --2- 172.21.15.29:0/1762047161 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b540731e0 0x7f5b54152550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c009dc0 tx=0x7f5b4c02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0da3b032b4a4a0d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:06.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.020+0000 7f5b52ffd640 1 -- 172.21.15.29:0/1762047161 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c039070 con 0x7f5b540731e0 2024-04-25T19:44:06.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.020+0000 7f5b52ffd640 1 -- 172.21.15.29:0/1762047161 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b4c02ccd0 con 0x7f5b540731e0 2024-04-25T19:44:06.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.020+0000 7f5b52ffd640 1 -- 172.21.15.29:0/1762047161 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c034660 con 0x7f5b540731e0 2024-04-25T19:44:06.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.020+0000 7f5b5a522640 1 -- 172.21.15.29:0/1762047161 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b540731e0 msgr2=0x7f5b54152550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:06.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.020+0000 7f5b5a522640 1 --2- 172.21.15.29:0/1762047161 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b540731e0 0x7f5b54152550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c009dc0 tx=0x7f5b4c02bf20 comp rx=0 tx=0).stop 2024-04-25T19:44:06.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.021+0000 7f5b5a522640 1 -- 172.21.15.29:0/1762047161 shutdown_connections 2024-04-25T19:44:06.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.021+0000 7f5b5a522640 1 --2- 172.21.15.29:0/1762047161 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b540731e0 0x7f5b54152550 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.021+0000 7f5b5a522640 1 -- 172.21.15.29:0/1762047161 >> 172.21.15.29:0/1762047161 conn(0x7f5b5406d6a0 msgr2=0x7f5b5406dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:06.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.021+0000 7f5b5a522640 1 -- 172.21.15.29:0/1762047161 shutdown_connections 2024-04-25T19:44:06.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.021+0000 7f5b5a522640 1 -- 172.21.15.29:0/1762047161 wait complete. 2024-04-25T19:44:06.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.022+0000 7f5b5a522640 1 Processor -- start 2024-04-25T19:44:06.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.022+0000 7f5b5a522640 1 -- start start 2024-04-25T19:44:06.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.023+0000 7f5b5a522640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b54072dd0 0x7f5b54144f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:06.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.023+0000 7f5b5a522640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b540731e0 0x7f5b541e0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:06.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.023+0000 7f5b5a522640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b540aae50 0x7f5b541e1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:06.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.023+0000 7f5b5a522640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5b54152ca0 con 0x7f5b540731e0 2024-04-25T19:44:06.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.023+0000 7f5b5a522640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5b54152e20 con 0x7f5b540aae50 2024-04-25T19:44:06.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.023+0000 7f5b5a522640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f5b54152fa0 con 0x7f5b54072dd0 2024-04-25T19:44:06.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.023+0000 7f5b53fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b54072dd0 0x7f5b54144f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:06.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.023+0000 7f5b537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b540731e0 0x7f5b541e0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:06.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.023+0000 7f5b53fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b54072dd0 0x7f5b54144f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:60318/0 (socket says 172.21.15.29:60318) 2024-04-25T19:44:06.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.023+0000 7f5b53fff640 1 -- 172.21.15.29:0/1167012674 learned_addr learned my addr 172.21.15.29:0/1167012674 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:06.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.023+0000 7f5b58abf640 1 --2- 172.21.15.29:0/1167012674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b540aae50 0x7f5b541e1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:06.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.024+0000 7f5b53fff640 1 -- 172.21.15.29:0/1167012674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b540aae50 msgr2=0x7f5b541e1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:06.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.024+0000 7f5b53fff640 1 --2- 172.21.15.29:0/1167012674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b540aae50 0x7f5b541e1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.024+0000 7f5b53fff640 1 -- 172.21.15.29:0/1167012674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b540731e0 msgr2=0x7f5b541e0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:06.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.024+0000 7f5b53fff640 1 --2- 172.21.15.29:0/1167012674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b540731e0 0x7f5b541e0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.024+0000 7f5b53fff640 1 -- 172.21.15.29:0/1167012674 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b54142890 con 0x7f5b54072dd0 2024-04-25T19:44:06.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.024+0000 7f5b537fe640 1 --2- 172.21.15.29:0/1167012674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b540731e0 0x7f5b541e0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:06.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.024+0000 7f5b53fff640 1 --2- 172.21.15.29:0/1167012674 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b54072dd0 0x7f5b54144f80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5b4400c3f0 tx=0x7f5b4400c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:06.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.024+0000 7f5b50ff9640 1 -- 172.21.15.29:0/1167012674 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b44017070 con 0x7f5b54072dd0 2024-04-25T19:44:06.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.024+0000 7f5b50ff9640 1 -- 172.21.15.29:0/1167012674 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b44005ce0 con 0x7f5b54072dd0 2024-04-25T19:44:06.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.024+0000 7f5b5a522640 1 -- 172.21.15.29:0/1167012674 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b54142b30 con 0x7f5b54072dd0 2024-04-25T19:44:06.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.024+0000 7f5b50ff9640 1 -- 172.21.15.29:0/1167012674 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b440072f0 con 0x7f5b54072dd0 2024-04-25T19:44:06.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.024+0000 7f5b5a522640 1 -- 172.21.15.29:0/1167012674 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b54141b20 con 0x7f5b54072dd0 2024-04-25T19:44:06.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.025+0000 7f5b5a522640 1 -- 172.21.15.29:0/1167012674 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b54069fe0 con 0x7f5b54072dd0 2024-04-25T19:44:06.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.026+0000 7f5b50ff9640 1 -- 172.21.15.29:0/1167012674 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f5b4401a070 con 0x7f5b54072dd0 2024-04-25T19:44:06.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.026+0000 7f5b50ff9640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:06.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.027+0000 7f5b50ff9640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:06.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.027+0000 7f5b50ff9640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:06.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.027+0000 7f5b50ff9640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:06.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.027+0000 7f5b50ff9640 1 --2- 172.21.15.29:0/1167012674 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5b400ac560 0x7f5b400aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:06.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.027+0000 7f5b50ff9640 1 -- 172.21.15.29:0/1167012674 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3482+0+0 (secure 0 0 0) 0x7f5b440d82e0 con 0x7f5b54072dd0 2024-04-25T19:44:06.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.027+0000 7f5b537fe640 1 --2- 172.21.15.29:0/1167012674 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5b400ac560 0x7f5b400aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:06.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.028+0000 7f5b537fe640 1 --2- 172.21.15.29:0/1167012674 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5b400ac560 0x7f5b400aea70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5b54147910 tx=0x7f5b48007de0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:06.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.031+0000 7f5b50ff9640 1 -- 172.21.15.29:0/1167012674 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5b4404eb80 con 0x7f5b54072dd0 2024-04-25T19:44:06.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.256+0000 7f5b5a522640 1 -- 172.21.15.29:0/1167012674 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b540507b0 con 0x7f5b54072dd0 2024-04-25T19:44:06.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.256+0000 7f5b50ff9640 1 -- 172.21.15.29:0/1167012674 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3040 (secure 0 0 0) 0x7f5b440a2bb0 con 0x7f5b54072dd0 2024-04-25T19:44:06.257 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:06.258 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6809/3582557422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3582557422},{"type":"v1","addr":"172.21.15.29:6809","nonce":3582557422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4234,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/1228868047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1228868047},{"type":"v1","addr":"172.21.15.102:6827","nonce":1228868047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4235,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6811/844243878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":844243878},{"type":"v1","addr":"172.21.15.29:6811","nonce":844243878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T19:44:06.258 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 3 2024-04-25T19:44:06.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.258+0000 7f5b5a522640 10 mgrc shutdown 2024-04-25T19:44:06.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.258+0000 7f5b5a522640 1 -- 172.21.15.29:0/1167012674 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5b400ac560 msgr2=0x7f5b400aea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:06.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.258+0000 7f5b5a522640 1 --2- 172.21.15.29:0/1167012674 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5b400ac560 0x7f5b400aea70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5b54147910 tx=0x7f5b48007de0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.258+0000 7f5b5a522640 1 -- 172.21.15.29:0/1167012674 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b54072dd0 msgr2=0x7f5b54144f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:06.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.258+0000 7f5b5a522640 1 --2- 172.21.15.29:0/1167012674 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b54072dd0 0x7f5b54144f80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5b4400c3f0 tx=0x7f5b4400c8b0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.258+0000 7f5b5a522640 1 -- 172.21.15.29:0/1167012674 shutdown_connections 2024-04-25T19:44:06.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.258+0000 7f5b5a522640 1 --2- 172.21.15.29:0/1167012674 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5b400ac560 0x7f5b400aea70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.258+0000 7f5b5a522640 1 --2- 172.21.15.29:0/1167012674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b540aae50 0x7f5b541e1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.258+0000 7f5b5a522640 1 --2- 172.21.15.29:0/1167012674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b540731e0 0x7f5b541e0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.258+0000 7f5b5a522640 1 --2- 172.21.15.29:0/1167012674 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b54072dd0 0x7f5b54144f80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.258+0000 7f5b5a522640 1 -- 172.21.15.29:0/1167012674 >> 172.21.15.29:0/1167012674 conn(0x7f5b5406d6a0 msgr2=0x7f5b5413c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:06.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.258+0000 7f5b5a522640 1 -- 172.21.15.29:0/1167012674 shutdown_connections 2024-04-25T19:44:06.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.258+0000 7f5b5a522640 1 -- 172.21.15.29:0/1167012674 wait complete. 2024-04-25T19:44:06.270 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:44:06.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.381+0000 7f453d710640 1 Processor -- start 2024-04-25T19:44:06.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.381+0000 7f453d710640 1 -- start start 2024-04-25T19:44:06.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.382+0000 7f453d710640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4538143e30 0x7f4538144240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:06.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.382+0000 7f453d710640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f45380acaa0 0x7f45380aceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:06.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.382+0000 7f453d710640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4538146c10 0x7f4538147020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:06.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.382+0000 7f453d710640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f4538147560 con 0x7f4538143e30 2024-04-25T19:44:06.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.382+0000 7f453d710640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f45380a5a90 con 0x7f4538146c10 2024-04-25T19:44:06.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.382+0000 7f453d710640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f4538147790 con 0x7f45380acaa0 2024-04-25T19:44:06.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.382+0000 7f4536ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4538143e30 0x7f4538144240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:06.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.382+0000 7f45367fc640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f45380acaa0 0x7f45380aceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:06.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.382+0000 7f4536ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4538143e30 0x7f4538144240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49722/0 (socket says 172.21.15.29:49722) 2024-04-25T19:44:06.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.382+0000 7f4536ffd640 1 -- 172.21.15.29:0/3877693200 learned_addr learned my addr 172.21.15.29:0/3877693200 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:06.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.382+0000 7f45367fc640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f45380acaa0 0x7f45380aceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:60330/0 (socket says 172.21.15.29:60330) 2024-04-25T19:44:06.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.383+0000 7f45377fe640 1 --2- 172.21.15.29:0/3877693200 >> v2:172.21.15.102:3300/0 conn(0x7f4538146c10 0x7f4538147020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:06.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.383+0000 7f4536ffd640 1 -- 172.21.15.29:0/3877693200 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f45380acaa0 msgr2=0x7f45380aceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:06.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.383+0000 7f4536ffd640 1 --2- 172.21.15.29:0/3877693200 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f45380acaa0 0x7f45380aceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.383+0000 7f4536ffd640 1 -- 172.21.15.29:0/3877693200 >> v2:172.21.15.102:3300/0 conn(0x7f4538146c10 msgr2=0x7f4538147020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:06.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.383+0000 7f4536ffd640 1 --2- 172.21.15.29:0/3877693200 >> v2:172.21.15.102:3300/0 conn(0x7f4538146c10 0x7f4538147020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.383+0000 7f4536ffd640 1 -- 172.21.15.29:0/3877693200 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4538147a50 con 0x7f4538143e30 2024-04-25T19:44:06.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.383+0000 7f45367fc640 1 --2- 172.21.15.29:0/3877693200 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f45380acaa0 0x7f45380aceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.384+0000 7f4536ffd640 1 --2- 172.21.15.29:0/3877693200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4538143e30 0x7f4538144240 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4530009870 tx=0x7f453002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839efb56e6943d0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.384+0000 7f4535ffb640 1 -- 172.21.15.29:0/3877693200 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4530035020 con 0x7f4538143e30 2024-04-25T19:44:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.384+0000 7f4535ffb640 1 -- 172.21.15.29:0/3877693200 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4530031640 con 0x7f4538143e30 2024-04-25T19:44:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.384+0000 7f4535ffb640 1 -- 172.21.15.29:0/3877693200 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453002d040 con 0x7f4538143e30 2024-04-25T19:44:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.384+0000 7f453d710640 1 -- 172.21.15.29:0/3877693200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4538143e30 msgr2=0x7f4538144240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.384+0000 7f453d710640 1 --2- 172.21.15.29:0/3877693200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4538143e30 0x7f4538144240 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4530009870 tx=0x7f453002b8f0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.384+0000 7f453d710640 1 -- 172.21.15.29:0/3877693200 shutdown_connections 2024-04-25T19:44:06.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.384+0000 7f453d710640 1 --2- 172.21.15.29:0/3877693200 >> v2:172.21.15.102:3300/0 conn(0x7f4538146c10 0x7f4538147020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.384+0000 7f453d710640 1 --2- 172.21.15.29:0/3877693200 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f45380acaa0 0x7f45380aceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.384+0000 7f453d710640 1 --2- 172.21.15.29:0/3877693200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4538143e30 0x7f4538144240 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.384+0000 7f453d710640 1 -- 172.21.15.29:0/3877693200 >> 172.21.15.29:0/3877693200 conn(0x7f45380aafc0 msgr2=0x7f45380ab3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:06.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.385+0000 7f453d710640 1 -- 172.21.15.29:0/3877693200 shutdown_connections 2024-04-25T19:44:06.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.385+0000 7f453d710640 1 -- 172.21.15.29:0/3877693200 wait complete. 2024-04-25T19:44:06.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.386+0000 7f453d710640 1 Processor -- start 2024-04-25T19:44:06.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.386+0000 7f453d710640 1 -- start start 2024-04-25T19:44:06.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.387+0000 7f453d710640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45380acaa0 0x7f45381d2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:06.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.387+0000 7f453d710640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4538143e30 0x7f4538145090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.387+0000 7f453d710640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4538146c10 0x7f4538142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.387+0000 7f453d710640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f453814a590 con 0x7f4538146c10 2024-04-25T19:44:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.387+0000 7f453d710640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f453814a410 con 0x7f45380acaa0 2024-04-25T19:44:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.387+0000 7f453d710640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f453814a710 con 0x7f4538143e30 2024-04-25T19:44:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.387+0000 7f45367fc640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4538143e30 0x7f4538145090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.387+0000 7f45367fc640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4538143e30 0x7f4538145090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:60336/0 (socket says 172.21.15.29:60336) 2024-04-25T19:44:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.387+0000 7f45377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4538146c10 0x7f4538142250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.387+0000 7f45367fc640 1 -- 172.21.15.29:0/3585877361 learned_addr learned my addr 172.21.15.29:0/3585877361 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.387+0000 7f4536ffd640 1 --2- 172.21.15.29:0/3585877361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45380acaa0 0x7f45381d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.387+0000 7f45367fc640 1 -- 172.21.15.29:0/3585877361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45380acaa0 msgr2=0x7f45381d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.387+0000 7f45367fc640 1 --2- 172.21.15.29:0/3585877361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45380acaa0 0x7f45381d2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.387+0000 7f45367fc640 1 -- 172.21.15.29:0/3585877361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4538146c10 msgr2=0x7f4538142250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.387+0000 7f45367fc640 1 --2- 172.21.15.29:0/3585877361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4538146c10 0x7f4538142250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.387+0000 7f45367fc640 1 -- 172.21.15.29:0/3585877361 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45381467d0 con 0x7f4538143e30 2024-04-25T19:44:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.388+0000 7f45367fc640 1 --2- 172.21.15.29:0/3585877361 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4538143e30 0x7f4538145090 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f452000e8e0 tx=0x7f452000eda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.388+0000 7f452f7fe640 1 -- 172.21.15.29:0/3585877361 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4520017070 con 0x7f4538143e30 2024-04-25T19:44:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.388+0000 7f452f7fe640 1 -- 172.21.15.29:0/3585877361 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f452000c8e0 con 0x7f4538143e30 2024-04-25T19:44:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.388+0000 7f453d710640 1 -- 172.21.15.29:0/3585877361 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f453809b1e0 con 0x7f4538143e30 2024-04-25T19:44:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.388+0000 7f452f7fe640 1 -- 172.21.15.29:0/3585877361 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f452000ccb0 con 0x7f4538143e30 2024-04-25T19:44:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.388+0000 7f453d710640 1 -- 172.21.15.29:0/3585877361 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4538144d90 con 0x7f4538143e30 2024-04-25T19:44:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.389+0000 7f453d710640 1 -- 172.21.15.29:0/3585877361 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45380507b0 con 0x7f4538143e30 2024-04-25T19:44:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.390+0000 7f452f7fe640 1 -- 172.21.15.29:0/3585877361 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f452001a070 con 0x7f4538143e30 2024-04-25T19:44:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.390+0000 7f452f7fe640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.391+0000 7f452f7fe640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.391+0000 7f452f7fe640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.391+0000 7f452f7fe640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.391+0000 7f452f7fe640 1 --2- 172.21.15.29:0/3585877361 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f45180ac560 0x7f45180aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.391+0000 7f452f7fe640 1 -- 172.21.15.29:0/3585877361 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3482+0+0 (secure 0 0 0) 0x7f45200d8ed0 con 0x7f4538143e30 2024-04-25T19:44:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.391+0000 7f4536ffd640 1 --2- 172.21.15.29:0/3585877361 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f45180ac560 0x7f45180aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:06.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.391+0000 7f4536ffd640 1 --2- 172.21.15.29:0/3585877361 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f45180ac560 0x7f45180aea70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4530036000 tx=0x7f453003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:06.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.395+0000 7f452f7fe640 1 -- 172.21.15.29:0/3585877361 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f45200a37a0 con 0x7f4538143e30 2024-04-25T19:44:06.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.579+0000 7f453d710640 1 -- 172.21.15.29:0/3585877361 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f45380bb530 con 0x7f4538143e30 2024-04-25T19:44:06.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.580+0000 7f452f7fe640 1 -- 172.21.15.29:0/3585877361 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+9734 (secure 0 0 0) 0x7f45200aae20 con 0x7f4538143e30 2024-04-25T19:44:06.581 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:06.581 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":16,"fsid":"1b3c39b5-e787-409e-b0bd-f4f491c89d72","created":"2024-04-25T19:43:47.854697+0000","modified":"2024-04-25T19:44:05.893192+0000","last_up_change":"2024-04-25T19:43:57.853237+0000","last_in_change":"2024-04-25T19:43:49.022776+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:43:58.096700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T19:44:03.617011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d805223-ee84-494f-bd31-59091906aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6801","nonce":695704147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6803","nonce":695704147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6807","nonce":695704147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6805","nonce":695704147}]},"public_addr":"172.21.15.29:6801/695704147","cluster_addr":"172.21.15.29:6803/695704147","heartbeat_back_addr":"172.21.15.29:6807/695704147","heartbeat_front_addr":"172.21.15.29:6805/695704147","state":["exists","up"]},{"osd":1,"uuid":"607c5038-38af-440f-8371-e71758549920","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6814","nonce":3135821452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6817","nonce":3135821452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6823","nonce":3135821452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6821","nonce":3135821452}]},"public_addr":"172.21.15.102:6814/3135821452","cluster_addr":"172.21.15.102:6817/3135821452","heartbeat_back_addr":"172.21.15.102:6823/3135821452","heartbeat_front_addr":"172.21.15.102:6821/3135821452","state":["exists","up"]},{"osd":2,"uuid":"6859ae76-54bb-4c55-b0dc-5e2735c0d31c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6801","nonce":2071844144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6803","nonce":2071844144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6811","nonce":2071844144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6806","nonce":2071844144}]},"public_addr":"172.21.15.102:6801/2071844144","cluster_addr":"172.21.15.102:6803/2071844144","heartbeat_back_addr":"172.21.15.102:6811/2071844144","heartbeat_front_addr":"172.21.15.102:6806/2071844144","state":["exists","up"]},{"osd":3,"uuid":"a98b7b88-5206-4f22-84a6-78212e0d5f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6805","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6807","nonce":339944984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6810","nonce":339944984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6820","nonce":339944984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6815","nonce":339944984}]},"public_addr":"172.21.15.102:6807/339944984","cluster_addr":"172.21.15.102:6810/339944984","heartbeat_back_addr":"172.21.15.102:6820/339944984","heartbeat_front_addr":"172.21.15.102:6815/339944984","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.121529+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.201155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.264387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.198713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:44:06.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.582+0000 7f453d710640 10 mgrc shutdown 2024-04-25T19:44:06.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.582+0000 7f453d710640 1 -- 172.21.15.29:0/3585877361 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f45180ac560 msgr2=0x7f45180aea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:06.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.582+0000 7f453d710640 1 --2- 172.21.15.29:0/3585877361 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f45180ac560 0x7f45180aea70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4530036000 tx=0x7f453003c040 comp rx=0 tx=0).stop 2024-04-25T19:44:06.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.582+0000 7f453d710640 1 -- 172.21.15.29:0/3585877361 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4538143e30 msgr2=0x7f4538145090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:06.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.582+0000 7f453d710640 1 --2- 172.21.15.29:0/3585877361 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4538143e30 0x7f4538145090 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f452000e8e0 tx=0x7f452000eda0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.582+0000 7f453d710640 1 -- 172.21.15.29:0/3585877361 shutdown_connections 2024-04-25T19:44:06.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.582+0000 7f453d710640 1 --2- 172.21.15.29:0/3585877361 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f45180ac560 0x7f45180aea70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.582+0000 7f453d710640 1 --2- 172.21.15.29:0/3585877361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4538146c10 0x7f4538142250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.582+0000 7f453d710640 1 --2- 172.21.15.29:0/3585877361 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4538143e30 0x7f4538145090 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.582+0000 7f453d710640 1 --2- 172.21.15.29:0/3585877361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45380acaa0 0x7f45381d2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.582+0000 7f453d710640 1 -- 172.21.15.29:0/3585877361 >> 172.21.15.29:0/3585877361 conn(0x7f45380aafc0 msgr2=0x7f45380684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:06.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.582+0000 7f453d710640 1 -- 172.21.15.29:0/3585877361 shutdown_connections 2024-04-25T19:44:06.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.582+0000 7f453d710640 1 -- 172.21.15.29:0/3585877361 wait complete. 2024-04-25T19:44:06.594 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:43:58.096700+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '13', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4, 'score_stable': 4, 'optimal_score': 0.5, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-25T19:44:03.617011+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T19:44:06.595 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:44:06.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.706+0000 7fa2e9a83640 1 Processor -- start 2024-04-25T19:44:06.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.706+0000 7fa2e9a83640 1 -- start start 2024-04-25T19:44:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.707+0000 7fa2e9a83640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa2e4144240 0x7fa2e4146c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.707+0000 7fa2e9a83640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2e4144650 con 0x7fa2e4143e30 2024-04-25T19:44:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.707+0000 7fa2e9a83640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2e40a5a90 con 0x7fa2e4147150 2024-04-25T19:44:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.707+0000 7fa2e9a83640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fa2e4147560 con 0x7fa2e4144240 2024-04-25T19:44:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.708+0000 7fa2e2ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa2e4144240 0x7fa2e4146c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.708+0000 7fa2e37fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa2e4143e30 0x7fa2e40acaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55430/0 (socket says 172.21.15.29:55430) 2024-04-25T19:44:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.708+0000 7fa2e2ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa2e4144240 0x7fa2e4146c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49738/0 (socket says 172.21.15.29:49738) 2024-04-25T19:44:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.708+0000 7fa2e37fe640 1 -- 172.21.15.29:0/2765549594 learned_addr learned my addr 172.21.15.29:0/2765549594 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.708+0000 7fa2e2ffd640 1 -- 172.21.15.29:0/2765549594 >> v1:172.21.15.102:6789/0 conn(0x7fa2e4147150 legacy=0x7fa2e4151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.708+0000 7fa2e2ffd640 1 -- 172.21.15.29:0/2765549594 >> v1:172.21.15.29:6789/0 conn(0x7fa2e4143e30 legacy=0x7fa2e40acaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:06.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.708+0000 7fa2e2ffd640 1 -- 172.21.15.29:0/2765549594 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e4147790 con 0x7fa2e4144240 2024-04-25T19:44:06.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.709+0000 7fa2e2ffd640 1 --2- 172.21.15.29:0/2765549594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e4144240 0x7fa2e4146c10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa2d80097a0 tx=0x7fa2d802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d18af3fcad12a78 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:06.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.709+0000 7fa2e27fc640 1 -- 172.21.15.29:0/2765549594 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8035020 con 0x7fa2e4144240 2024-04-25T19:44:06.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.709+0000 7fa2e27fc640 1 -- 172.21.15.29:0/2765549594 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2d8031830 con 0x7fa2e4144240 2024-04-25T19:44:06.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.709+0000 7fa2e27fc640 1 -- 172.21.15.29:0/2765549594 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8031c00 con 0x7fa2e4144240 2024-04-25T19:44:06.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.709+0000 7fa2e9a83640 1 -- 172.21.15.29:0/2765549594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e4144240 msgr2=0x7fa2e4146c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:06.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.709+0000 7fa2e9a83640 1 --2- 172.21.15.29:0/2765549594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e4144240 0x7fa2e4146c10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa2d80097a0 tx=0x7fa2d802b8d0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.710+0000 7fa2e9a83640 1 -- 172.21.15.29:0/2765549594 shutdown_connections 2024-04-25T19:44:06.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.710+0000 7fa2e9a83640 1 --2- 172.21.15.29:0/2765549594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e4144240 0x7fa2e4146c10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.710+0000 7fa2e9a83640 1 -- 172.21.15.29:0/2765549594 >> 172.21.15.29:0/2765549594 conn(0x7fa2e40aafc0 msgr2=0x7fa2e40ab3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:06.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.710+0000 7fa2e9a83640 1 -- 172.21.15.29:0/2765549594 shutdown_connections 2024-04-25T19:44:06.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.710+0000 7fa2e9a83640 1 -- 172.21.15.29:0/2765549594 wait complete. 2024-04-25T19:44:06.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.711+0000 7fa2e9a83640 1 Processor -- start 2024-04-25T19:44:06.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.711+0000 7fa2e9a83640 1 -- start start 2024-04-25T19:44:06.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.712+0000 7fa2e9a83640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e4143e30 0x7fa2e414bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:06.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.712+0000 7fa2e9a83640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e4144240 0x7fa2e4140ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:06.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.712+0000 7fa2e9a83640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa2e4147150 0x7fa2e41e0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:06.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.712+0000 7fa2e9a83640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e40ad330 con 0x7fa2e4144240 2024-04-25T19:44:06.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.712+0000 7fa2e9a83640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e40ad1b0 con 0x7fa2e4143e30 2024-04-25T19:44:06.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.712+0000 7fa2e9a83640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fa2e4152cb0 con 0x7fa2e4147150 2024-04-25T19:44:06.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.712+0000 7fa2e2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e4144240 0x7fa2e4140ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:06.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.712+0000 7fa2e37fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e4143e30 0x7fa2e414bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:06.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.712+0000 7fa2e2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e4144240 0x7fa2e4140ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49750/0 (socket says 172.21.15.29:49750) 2024-04-25T19:44:06.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.712+0000 7fa2e2ffd640 1 -- 172.21.15.29:0/1618982274 learned_addr learned my addr 172.21.15.29:0/1618982274 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:06.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.712+0000 7fa2e3fff640 1 --2- 172.21.15.29:0/1618982274 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa2e4147150 0x7fa2e41e0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:06.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.713+0000 7fa2e2ffd640 1 -- 172.21.15.29:0/1618982274 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa2e4147150 msgr2=0x7fa2e41e0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:06.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.713+0000 7fa2e2ffd640 1 --2- 172.21.15.29:0/1618982274 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa2e4147150 0x7fa2e41e0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.713+0000 7fa2e2ffd640 1 -- 172.21.15.29:0/1618982274 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e4143e30 msgr2=0x7fa2e414bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:06.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.713+0000 7fa2e2ffd640 1 --2- 172.21.15.29:0/1618982274 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e4143e30 0x7fa2e414bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.713+0000 7fa2e2ffd640 1 -- 172.21.15.29:0/1618982274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e414c250 con 0x7fa2e4144240 2024-04-25T19:44:06.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.713+0000 7fa2e2ffd640 1 --2- 172.21.15.29:0/1618982274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e4144240 0x7fa2e4140ed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa2d80098c0 tx=0x7fa2d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:06.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.713+0000 7fa2cbfff640 1 -- 172.21.15.29:0/1618982274 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8040070 con 0x7fa2e4144240 2024-04-25T19:44:06.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.713+0000 7fa2cbfff640 1 -- 172.21.15.29:0/1618982274 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2d8039a20 con 0x7fa2e4144240 2024-04-25T19:44:06.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.713+0000 7fa2e9a83640 1 -- 172.21.15.29:0/1618982274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2e4141410 con 0x7fa2e4144240 2024-04-25T19:44:06.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.713+0000 7fa2cbfff640 1 -- 172.21.15.29:0/1618982274 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8035040 con 0x7fa2e4144240 2024-04-25T19:44:06.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.713+0000 7fa2e9a83640 1 -- 172.21.15.29:0/1618982274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2e4139cd0 con 0x7fa2e4144240 2024-04-25T19:44:06.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.714+0000 7fa2e9a83640 1 -- 172.21.15.29:0/1618982274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2e40507b0 con 0x7fa2e4144240 2024-04-25T19:44:06.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.715+0000 7fa2cbfff640 1 -- 172.21.15.29:0/1618982274 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7fa2d8038020 con 0x7fa2e4144240 2024-04-25T19:44:06.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.715+0000 7fa2cbfff640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:06.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.716+0000 7fa2cbfff640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:06.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.716+0000 7fa2cbfff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:06.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.716+0000 7fa2cbfff640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:06.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.716+0000 7fa2cbfff640 1 --2- 172.21.15.29:0/1618982274 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fa2d00ac530 0x7fa2d00aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:06.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.716+0000 7fa2cbfff640 1 -- 172.21.15.29:0/1618982274 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3482+0+0 (secure 0 0 0) 0x7fa2d80fad00 con 0x7fa2e4144240 2024-04-25T19:44:06.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.716+0000 7fa2e37fe640 1 --2- 172.21.15.29:0/1618982274 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fa2d00ac530 0x7fa2d00aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:06.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.717+0000 7fa2e37fe640 1 --2- 172.21.15.29:0/1618982274 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fa2d00ac530 0x7fa2d00aea40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4005880 tx=0x7fa2d4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:06.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.719+0000 7fa2cbfff640 1 -- 172.21.15.29:0/1618982274 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa2d807fb80 con 0x7fa2e4144240 2024-04-25T19:44:06.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.904+0000 7fa2e9a83640 1 -- 172.21.15.29:0/1618982274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa2e4141c80 con 0x7fa2e4144240 2024-04-25T19:44:06.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.904+0000 7fa2cbfff640 1 -- 172.21.15.29:0/1618982274 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7fa2d80c5cc0 con 0x7fa2e4144240 2024-04-25T19:44:06.905 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-04-25T19:44:06.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.906+0000 7fa2e9a83640 10 mgrc shutdown 2024-04-25T19:44:06.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.906+0000 7fa2e9a83640 1 -- 172.21.15.29:0/1618982274 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fa2d00ac530 msgr2=0x7fa2d00aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:06.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.906+0000 7fa2e9a83640 1 --2- 172.21.15.29:0/1618982274 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fa2d00ac530 0x7fa2d00aea40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4005880 tx=0x7fa2d4009040 comp rx=0 tx=0).stop 2024-04-25T19:44:06.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.906+0000 7fa2e9a83640 1 -- 172.21.15.29:0/1618982274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e4144240 msgr2=0x7fa2e4140ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:06.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.906+0000 7fa2e9a83640 1 --2- 172.21.15.29:0/1618982274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e4144240 0x7fa2e4140ed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa2d80098c0 tx=0x7fa2d80046c0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.907+0000 7fa2e9a83640 1 -- 172.21.15.29:0/1618982274 shutdown_connections 2024-04-25T19:44:06.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.907+0000 7fa2e9a83640 1 --2- 172.21.15.29:0/1618982274 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fa2d00ac530 0x7fa2d00aea40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.907+0000 7fa2e9a83640 1 --2- 172.21.15.29:0/1618982274 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa2e4147150 0x7fa2e41e0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.907+0000 7fa2e9a83640 1 --2- 172.21.15.29:0/1618982274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e4144240 0x7fa2e4140ed0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.907+0000 7fa2e9a83640 1 --2- 172.21.15.29:0/1618982274 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e4143e30 0x7fa2e414bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:06.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.907+0000 7fa2e9a83640 1 -- 172.21.15.29:0/1618982274 >> 172.21.15.29:0/1618982274 conn(0x7fa2e40aafc0 msgr2=0x7fa2e4141ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:06.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.907+0000 7fa2e9a83640 1 -- 172.21.15.29:0/1618982274 shutdown_connections 2024-04-25T19:44:06.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:06.907+0000 7fa2e9a83640 1 -- 172.21.15.29:0/1618982274 wait complete. 2024-04-25T19:44:06.919 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-04-25T19:44:07.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.034+0000 7f6ffddd8640 1 Processor -- start 2024-04-25T19:44:07.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.034+0000 7f6ffddd8640 1 -- start start 2024-04-25T19:44:07.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.035+0000 7f6ffddd8640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ff806da90 0x7f6ff80a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.035+0000 7f6ffddd8640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6ff80ad450 0x7f6ff81476e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.035+0000 7f6ffddd8640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff8146c60 con 0x7f6ff81472d0 2024-04-25T19:44:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.035+0000 7f6ffddd8640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f6ff80a5c10 con 0x7f6ff80ad450 2024-04-25T19:44:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.035+0000 7f6ffddd8640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f6ff80ad9d0 con 0x7f6ff806da90 2024-04-25T19:44:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.035+0000 7f6ff77fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ff806da90 0x7f6ff80a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.035+0000 7f6ff77fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ff806da90 0x7f6ff80a56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:60368/0 (socket says 172.21.15.29:60368) 2024-04-25T19:44:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.035+0000 7f6ff77fe640 1 -- 172.21.15.29:0/1335750175 learned_addr learned my addr 172.21.15.29:0/1335750175 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.035+0000 7f6ff77fe640 1 -- 172.21.15.29:0/1335750175 >> v2:172.21.15.102:3300/0 conn(0x7f6ff80ad450 msgr2=0x7f6ff81476e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.035+0000 7f6ff77fe640 1 --2- 172.21.15.29:0/1335750175 >> v2:172.21.15.102:3300/0 conn(0x7f6ff80ad450 0x7f6ff81476e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.035+0000 7f6ff77fe640 1 -- 172.21.15.29:0/1335750175 >> v1:172.21.15.102:6789/0 conn(0x7f6ff81472d0 legacy=0x7f6ff80acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.035+0000 7f6ff77fe640 1 -- 172.21.15.29:0/1335750175 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ff8152400 con 0x7f6ff806da90 2024-04-25T19:44:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.036+0000 7f6ff77fe640 1 --2- 172.21.15.29:0/1335750175 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ff806da90 0x7f6ff80a56d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009870 tx=0x7f6fe802b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edbdb1914ab68e8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.036+0000 7f6ff67fc640 1 -- 172.21.15.29:0/1335750175 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8035020 con 0x7f6ff806da90 2024-04-25T19:44:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.036+0000 7f6ff67fc640 1 -- 172.21.15.29:0/1335750175 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fe8031870 con 0x7f6ff806da90 2024-04-25T19:44:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.036+0000 7f6ff67fc640 1 -- 172.21.15.29:0/1335750175 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8031c40 con 0x7f6ff806da90 2024-04-25T19:44:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.037+0000 7f6ffddd8640 1 -- 172.21.15.29:0/1335750175 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ff806da90 msgr2=0x7f6ff80a56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.037+0000 7f6ffddd8640 1 --2- 172.21.15.29:0/1335750175 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ff806da90 0x7f6ff80a56d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009870 tx=0x7f6fe802b910 comp rx=0 tx=0).stop 2024-04-25T19:44:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.037+0000 7f6ffddd8640 1 -- 172.21.15.29:0/1335750175 shutdown_connections 2024-04-25T19:44:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.037+0000 7f6ffddd8640 1 --2- 172.21.15.29:0/1335750175 >> v2:172.21.15.102:3300/0 conn(0x7f6ff80ad450 0x7f6ff81476e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.037+0000 7f6ffddd8640 1 --2- 172.21.15.29:0/1335750175 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ff806da90 0x7f6ff80a56d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.037+0000 7f6ffddd8640 1 -- 172.21.15.29:0/1335750175 >> 172.21.15.29:0/1335750175 conn(0x7f6ff80bc3b0 msgr2=0x7f6ff80bc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.037+0000 7f6ffddd8640 1 -- 172.21.15.29:0/1335750175 shutdown_connections 2024-04-25T19:44:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.037+0000 7f6ffddd8640 1 -- 172.21.15.29:0/1335750175 wait complete. 2024-04-25T19:44:07.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.038+0000 7f6ffddd8640 1 Processor -- start 2024-04-25T19:44:07.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.038+0000 7f6ffddd8640 1 -- start start 2024-04-25T19:44:07.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.039+0000 7f6ffddd8640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ff806da90 0x7f6ff8134fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:07.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.039+0000 7f6ffddd8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff80ad450 0x7f6ff8149800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:07.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.039+0000 7f6ffddd8640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ff81472d0 0x7f6ff81481a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:07.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.039+0000 7f6ffddd8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff81517c0 con 0x7f6ff80ad450 2024-04-25T19:44:07.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.039+0000 7f6ffddd8640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff8151640 con 0x7f6ff81472d0 2024-04-25T19:44:07.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.039+0000 7f6ffddd8640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f6ff8151940 con 0x7f6ff806da90 2024-04-25T19:44:07.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.040+0000 7f6ff77fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ff806da90 0x7f6ff8134fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:07.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.040+0000 7f6ff6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff80ad450 0x7f6ff8149800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:07.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.040+0000 7f6ff77fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ff806da90 0x7f6ff8134fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:60378/0 (socket says 172.21.15.29:60378) 2024-04-25T19:44:07.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.040+0000 7f6ff77fe640 1 -- 172.21.15.29:0/1807615098 learned_addr learned my addr 172.21.15.29:0/1807615098 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:07.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.040+0000 7f6ff7fff640 1 --2- 172.21.15.29:0/1807615098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ff81472d0 0x7f6ff81481a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:07.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.040+0000 7f6ff77fe640 1 -- 172.21.15.29:0/1807615098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ff81472d0 msgr2=0x7f6ff81481a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:07.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.040+0000 7f6ff77fe640 1 --2- 172.21.15.29:0/1807615098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ff81472d0 0x7f6ff81481a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.040+0000 7f6ff77fe640 1 -- 172.21.15.29:0/1807615098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff80ad450 msgr2=0x7f6ff8149800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:07.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.040+0000 7f6ff77fe640 1 --2- 172.21.15.29:0/1807615098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff80ad450 0x7f6ff8149800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.040+0000 7f6ff77fe640 1 -- 172.21.15.29:0/1807615098 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ff8135600 con 0x7f6ff806da90 2024-04-25T19:44:07.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.040+0000 7f6ff6ffd640 1 --2- 172.21.15.29:0/1807615098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff80ad450 0x7f6ff8149800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:07.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.040+0000 7f6ff77fe640 1 --2- 172.21.15.29:0/1807615098 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ff806da90 0x7f6ff8134fe0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8000c00 tx=0x7f6fe80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:07.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.040+0000 7f6fdffff640 1 -- 172.21.15.29:0/1807615098 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe803ad60 con 0x7f6ff806da90 2024-04-25T19:44:07.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.040+0000 7f6fdffff640 1 -- 172.21.15.29:0/1807615098 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fe80443f0 con 0x7f6ff806da90 2024-04-25T19:44:07.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.040+0000 7f6ffddd8640 1 -- 172.21.15.29:0/1807615098 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ff8149d40 con 0x7f6ff806da90 2024-04-25T19:44:07.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.040+0000 7f6fdffff640 1 -- 172.21.15.29:0/1807615098 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe80447c0 con 0x7f6ff806da90 2024-04-25T19:44:07.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.040+0000 7f6ffddd8640 1 -- 172.21.15.29:0/1807615098 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ff81488c0 con 0x7f6ff806da90 2024-04-25T19:44:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.042+0000 7f6ffddd8640 1 -- 172.21.15.29:0/1807615098 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ff8148bb0 con 0x7f6ff806da90 2024-04-25T19:44:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.043+0000 7f6fdffff640 1 -- 172.21.15.29:0/1807615098 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f6fe804b060 con 0x7f6ff806da90 2024-04-25T19:44:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.043+0000 7f6fdffff640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.043+0000 7f6fdffff640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.043+0000 7f6fdffff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.043+0000 7f6fdffff640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.043+0000 7f6fdffff640 1 --2- 172.21.15.29:0/1807615098 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f6fe40ac530 0x7f6fe40aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.043+0000 7f6fdffff640 1 -- 172.21.15.29:0/1807615098 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3482+0+0 (secure 0 0 0) 0x7f6fe80fb1f0 con 0x7f6ff806da90 2024-04-25T19:44:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.044+0000 7f6ff6ffd640 1 --2- 172.21.15.29:0/1807615098 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f6fe40ac530 0x7f6fe40aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:07.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.044+0000 7f6ff6ffd640 1 --2- 172.21.15.29:0/1807615098 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f6fe40ac530 0x7f6fe40aea40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8134d80 tx=0x7f6fec009060 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.048+0000 7f6fdffff640 1 -- 172.21.15.29:0/1807615098 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6ff8148bb0 con 0x7f6ff806da90 2024-04-25T19:44:07.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.230+0000 7f6ffddd8640 1 -- 172.21.15.29:0/1807615098 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7f6ff80507b0 con 0x7f6ff806da90 2024-04-25T19:44:07.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.231+0000 7f6fdffff640 1 -- 172.21.15.29:0/1807615098 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v16) ==== 92+0+10 (secure 0 0 0) 0x7f6fe803f070 con 0x7f6ff806da90 2024-04-25T19:44:07.232 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 8 2024-04-25T19:44:07.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.232+0000 7f6ffddd8640 10 mgrc shutdown 2024-04-25T19:44:07.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.232+0000 7f6ffddd8640 1 -- 172.21.15.29:0/1807615098 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f6fe40ac530 msgr2=0x7f6fe40aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:07.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.232+0000 7f6ffddd8640 1 --2- 172.21.15.29:0/1807615098 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f6fe40ac530 0x7f6fe40aea40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8134d80 tx=0x7f6fec009060 comp rx=0 tx=0).stop 2024-04-25T19:44:07.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.233+0000 7f6ffddd8640 1 -- 172.21.15.29:0/1807615098 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ff806da90 msgr2=0x7f6ff8134fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:07.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.233+0000 7f6ffddd8640 1 --2- 172.21.15.29:0/1807615098 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ff806da90 0x7f6ff8134fe0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8000c00 tx=0x7f6fe80046c0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.233+0000 7f6ffddd8640 1 -- 172.21.15.29:0/1807615098 shutdown_connections 2024-04-25T19:44:07.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.233+0000 7f6ffddd8640 1 --2- 172.21.15.29:0/1807615098 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f6fe40ac530 0x7f6fe40aea40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.233+0000 7f6ffddd8640 1 --2- 172.21.15.29:0/1807615098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ff81472d0 0x7f6ff81481a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.233+0000 7f6ffddd8640 1 --2- 172.21.15.29:0/1807615098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff80ad450 0x7f6ff8149800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.233+0000 7f6ffddd8640 1 --2- 172.21.15.29:0/1807615098 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ff806da90 0x7f6ff8134fe0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.233+0000 7f6ffddd8640 1 -- 172.21.15.29:0/1807615098 >> 172.21.15.29:0/1807615098 conn(0x7f6ff80bc3b0 msgr2=0x7f6ff80aaf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:07.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.233+0000 7f6ffddd8640 1 -- 172.21.15.29:0/1807615098 shutdown_connections 2024-04-25T19:44:07.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.233+0000 7f6ffddd8640 1 -- 172.21.15.29:0/1807615098 wait complete. 2024-04-25T19:44:07.246 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T19:44:07.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.356+0000 7fd06b23a640 1 Processor -- start 2024-04-25T19:44:07.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.356+0000 7fd06b23a640 1 -- start start 2024-04-25T19:44:07.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.357+0000 7fd06b23a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd064150590 0x7fd0641529b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:07.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.357+0000 7fd06b23a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd064144650 con 0x7fd064144240 2024-04-25T19:44:07.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.357+0000 7fd06b23a640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0640a5a90 con 0x7fd064143e30 2024-04-25T19:44:07.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.357+0000 7fd06b23a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fd064147790 con 0x7fd064150590 2024-04-25T19:44:07.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.357+0000 7fd0697d7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd064150590 0x7fd0641529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:07.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.357+0000 7fd063fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd064144240 0x7fd064146c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55436/0 (socket says 172.21.15.29:55436) 2024-04-25T19:44:07.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.357+0000 7fd063fff640 1 -- 172.21.15.29:0/2526611492 learned_addr learned my addr 172.21.15.29:0/2526611492 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:07.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.357+0000 7fd0697d7640 1 -- 172.21.15.29:0/2526611492 >> v1:172.21.15.102:6789/0 conn(0x7fd064143e30 legacy=0x7fd0640acaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:07.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.357+0000 7fd0697d7640 1 -- 172.21.15.29:0/2526611492 >> v1:172.21.15.29:6789/0 conn(0x7fd064144240 legacy=0x7fd064146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:07.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.357+0000 7fd0637fe640 1 -- 172.21.15.29:0/2526611492 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 379899436 0 0) 0x7fd050001110 con 0x7fd064144240 2024-04-25T19:44:07.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.357+0000 7fd0697d7640 1 -- 172.21.15.29:0/2526611492 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd064147a50 con 0x7fd064150590 2024-04-25T19:44:07.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.358+0000 7fd0697d7640 1 --2- 172.21.15.29:0/2526611492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd064150590 0x7fd0641529b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd05c009d90 tx=0x7fd05c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8316c0ea8980587 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:07.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.358+0000 7fd0637fe640 1 -- 172.21.15.29:0/2526611492 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c039070 con 0x7fd064150590 2024-04-25T19:44:07.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.358+0000 7fd0637fe640 1 -- 172.21.15.29:0/2526611492 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd05c02cc60 con 0x7fd064150590 2024-04-25T19:44:07.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.358+0000 7fd0637fe640 1 -- 172.21.15.29:0/2526611492 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c034660 con 0x7fd064150590 2024-04-25T19:44:07.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.359+0000 7fd06b23a640 1 -- 172.21.15.29:0/2526611492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd064150590 msgr2=0x7fd0641529b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:07.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.359+0000 7fd06b23a640 1 --2- 172.21.15.29:0/2526611492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd064150590 0x7fd0641529b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd05c009d90 tx=0x7fd05c02beb0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.359+0000 7fd06b23a640 1 -- 172.21.15.29:0/2526611492 shutdown_connections 2024-04-25T19:44:07.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.359+0000 7fd06b23a640 1 --2- 172.21.15.29:0/2526611492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd064150590 0x7fd0641529b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.359+0000 7fd06b23a640 1 -- 172.21.15.29:0/2526611492 >> 172.21.15.29:0/2526611492 conn(0x7fd0640aafc0 msgr2=0x7fd0640ab3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:07.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.359+0000 7fd06b23a640 1 -- 172.21.15.29:0/2526611492 shutdown_connections 2024-04-25T19:44:07.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.359+0000 7fd06b23a640 1 -- 172.21.15.29:0/2526611492 wait complete. 2024-04-25T19:44:07.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.360+0000 7fd06b23a640 1 Processor -- start 2024-04-25T19:44:07.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.360+0000 7fd06b23a640 1 -- start start 2024-04-25T19:44:07.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd06b23a640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd064143e30 0x7fd064137ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:07.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd06b23a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd064144240 0x7fd06413ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:07.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd06b23a640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd064150590 0x7fd0640a7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:07.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd06b23a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd0641474a0 con 0x7fd064144240 2024-04-25T19:44:07.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd06b23a640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd064147320 con 0x7fd064150590 2024-04-25T19:44:07.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd06b23a640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fd064152ef0 con 0x7fd064143e30 2024-04-25T19:44:07.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd063fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd064144240 0x7fd06413ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:07.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd068fd6640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd064143e30 0x7fd064137ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:07.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd063fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd064144240 0x7fd06413ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49786/0 (socket says 172.21.15.29:49786) 2024-04-25T19:44:07.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd063fff640 1 -- 172.21.15.29:0/1971992135 learned_addr learned my addr 172.21.15.29:0/1971992135 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:07.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd068fd6640 1 --2- 172.21.15.29:0/1971992135 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd064143e30 0x7fd064137ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:60392/0 (socket says 172.21.15.29:60392) 2024-04-25T19:44:07.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd0697d7640 1 --2- 172.21.15.29:0/1971992135 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd064150590 0x7fd0640a7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:07.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd063fff640 1 -- 172.21.15.29:0/1971992135 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd064143e30 msgr2=0x7fd064137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:07.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd063fff640 1 --2- 172.21.15.29:0/1971992135 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd064143e30 0x7fd064137ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd063fff640 1 -- 172.21.15.29:0/1971992135 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd064150590 msgr2=0x7fd0640a7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:07.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd063fff640 1 --2- 172.21.15.29:0/1971992135 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd064150590 0x7fd0640a7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd063fff640 1 -- 172.21.15.29:0/1971992135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0641524d0 con 0x7fd064144240 2024-04-25T19:44:07.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd0697d7640 1 --2- 172.21.15.29:0/1971992135 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd064150590 0x7fd0640a7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:44:07.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd063fff640 1 --2- 172.21.15.29:0/1971992135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd064144240 0x7fd06413ca00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd05000c420 tx=0x7fd05000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:07.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd0617fa640 1 -- 172.21.15.29:0/1971992135 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0500074e0 con 0x7fd064144240 2024-04-25T19:44:07.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd0617fa640 1 -- 172.21.15.29:0/1971992135 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0500047e0 con 0x7fd064144240 2024-04-25T19:44:07.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd0617fa640 1 -- 172.21.15.29:0/1971992135 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd050004c10 con 0x7fd064144240 2024-04-25T19:44:07.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd06b23a640 1 -- 172.21.15.29:0/1971992135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0640a7770 con 0x7fd064144240 2024-04-25T19:44:07.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.362+0000 7fd06b23a640 1 -- 172.21.15.29:0/1971992135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0640a0b20 con 0x7fd064144240 2024-04-25T19:44:07.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.364+0000 7fd06b23a640 1 -- 172.21.15.29:0/1971992135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0641378f0 con 0x7fd064144240 2024-04-25T19:44:07.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.365+0000 7fd0617fa640 1 -- 172.21.15.29:0/1971992135 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7fd050011070 con 0x7fd064144240 2024-04-25T19:44:07.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.365+0000 7fd0617fa640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:07.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.365+0000 7fd0617fa640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:07.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.365+0000 7fd0617fa640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:07.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.365+0000 7fd0617fa640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:07.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.365+0000 7fd0617fa640 1 --2- 172.21.15.29:0/1971992135 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fd0480ac4b0 0x7fd0480ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:07.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.366+0000 7fd0617fa640 1 -- 172.21.15.29:0/1971992135 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3482+0+0 (secure 0 0 0) 0x7fd0500d9300 con 0x7fd064144240 2024-04-25T19:44:07.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.366+0000 7fd068fd6640 1 --2- 172.21.15.29:0/1971992135 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fd0480ac4b0 0x7fd0480ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:07.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.366+0000 7fd068fd6640 1 --2- 172.21.15.29:0/1971992135 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fd0480ac4b0 0x7fd0480ae9c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd058000f60 tx=0x7fd058009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:07.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.369+0000 7fd0617fa640 1 -- 172.21.15.29:0/1971992135 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd0641378f0 con 0x7fd064144240 2024-04-25T19:44:07.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.553+0000 7fd06b23a640 1 -- 172.21.15.29:0/1971992135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fd0640507b0 con 0x7fd064144240 2024-04-25T19:44:07.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.554+0000 7fd0617fa640 1 -- 172.21.15.29:0/1971992135 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+126 (secure 0 0 0) 0x7fd064144650 con 0x7fd064144240 2024-04-25T19:44:07.557 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:07.557 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2024-04-25T19:44:07.557 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-04-25T19:44:07.557 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2024-04-25T19:44:07.557 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2024-04-25T19:44:07.557 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-25T19:44:07.557 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-04-25T19:44:07.558 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 2, 2024-04-25T19:44:07.558 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": "rbd" 2024-04-25T19:44:07.558 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-25T19:44:07.558 INFO:teuthology.orchestra.run.smithi029.stdout:] 2024-04-25T19:44:07.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.556+0000 7fd06b23a640 10 mgrc shutdown 2024-04-25T19:44:07.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.556+0000 7fd06b23a640 1 -- 172.21.15.29:0/1971992135 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fd0480ac4b0 msgr2=0x7fd0480ae9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:07.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.556+0000 7fd06b23a640 1 --2- 172.21.15.29:0/1971992135 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fd0480ac4b0 0x7fd0480ae9c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd058000f60 tx=0x7fd058009040 comp rx=0 tx=0).stop 2024-04-25T19:44:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.556+0000 7fd06b23a640 1 -- 172.21.15.29:0/1971992135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd064144240 msgr2=0x7fd06413ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.556+0000 7fd06b23a640 1 --2- 172.21.15.29:0/1971992135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd064144240 0x7fd06413ca00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd05000c420 tx=0x7fd05000c8e0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.556+0000 7fd06b23a640 1 -- 172.21.15.29:0/1971992135 shutdown_connections 2024-04-25T19:44:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.556+0000 7fd06b23a640 1 --2- 172.21.15.29:0/1971992135 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fd0480ac4b0 0x7fd0480ae9c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.556+0000 7fd06b23a640 1 --2- 172.21.15.29:0/1971992135 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd064150590 0x7fd0640a7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.556+0000 7fd06b23a640 1 --2- 172.21.15.29:0/1971992135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd064144240 0x7fd06413ca00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.556+0000 7fd06b23a640 1 --2- 172.21.15.29:0/1971992135 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd064143e30 0x7fd064137ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.556+0000 7fd06b23a640 1 -- 172.21.15.29:0/1971992135 >> 172.21.15.29:0/1971992135 conn(0x7fd0640aafc0 msgr2=0x7fd064146a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.556+0000 7fd06b23a640 1 -- 172.21.15.29:0/1971992135 shutdown_connections 2024-04-25T19:44:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.556+0000 7fd06b23a640 1 -- 172.21.15.29:0/1971992135 wait complete. 2024-04-25T19:44:07.569 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T19:44:07.569 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T19:44:07.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.682+0000 7f06b3dac640 1 Processor -- start 2024-04-25T19:44:07.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.682+0000 7f06b3dac640 1 -- start start 2024-04-25T19:44:07.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.683+0000 7f06b3dac640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f06ac0a55d0 0x7f06ac146140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:07.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.683+0000 7f06b3dac640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f06ac146680 0x7f06ac1513b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:07.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.683+0000 7f06b3dac640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06ac0ad300 con 0x7f06ac0ae260 2024-04-25T19:44:07.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.683+0000 7f06b3dac640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f06ac0a5ff0 con 0x7f06ac146680 2024-04-25T19:44:07.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.683+0000 7f06b3dac640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f06ac0ad5c0 con 0x7f06ac0a55d0 2024-04-25T19:44:07.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.683+0000 7f06b2349640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f06ac146680 0x7f06ac1513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:07.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.683+0000 7f06b1347640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f06ac0a55d0 0x7f06ac146140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:07.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.683+0000 7f06b2349640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f06ac146680 0x7f06ac1513b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36408/0 (socket says 172.21.15.29:36408) 2024-04-25T19:44:07.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.683+0000 7f06b1347640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f06ac0a55d0 0x7f06ac146140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34294/0 (socket says 172.21.15.29:34294) 2024-04-25T19:44:07.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.683+0000 7f06b2349640 1 -- 172.21.15.29:0/3782538352 learned_addr learned my addr 172.21.15.29:0/3782538352 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:07.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.684+0000 7f06b2349640 1 -- 172.21.15.29:0/3782538352 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f06ac0a55d0 msgr2=0x7f06ac146140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:07.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.684+0000 7f06b2349640 1 --2- 172.21.15.29:0/3782538352 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f06ac0a55d0 0x7f06ac146140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.684+0000 7f06b2349640 1 -- 172.21.15.29:0/3782538352 >> v1:172.21.15.102:6789/0 conn(0x7f06ac0ae260 legacy=0x7f06ac0acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:07.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.684+0000 7f06b2349640 1 -- 172.21.15.29:0/3782538352 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06ac0aad10 con 0x7f06ac146680 2024-04-25T19:44:07.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.684+0000 7f06b1347640 1 --2- 172.21.15.29:0/3782538352 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f06ac0a55d0 0x7f06ac146140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:07.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.684+0000 7f06b2349640 1 --2- 172.21.15.29:0/3782538352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06ac146680 0x7f06ac1513b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f06a8009e50 tx=0x7f06a802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d3b40686516c967 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:07.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.685+0000 7f06b0b46640 1 -- 172.21.15.29:0/3782538352 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a802caf0 con 0x7f06ac146680 2024-04-25T19:44:07.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.685+0000 7f06b0b46640 1 -- 172.21.15.29:0/3782538352 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06a802cc90 con 0x7f06ac146680 2024-04-25T19:44:07.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.685+0000 7f06b0b46640 1 -- 172.21.15.29:0/3782538352 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a8034610 con 0x7f06ac146680 2024-04-25T19:44:07.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.685+0000 7f06b3dac640 1 -- 172.21.15.29:0/3782538352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06ac146680 msgr2=0x7f06ac1513b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:07.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.685+0000 7f06b3dac640 1 --2- 172.21.15.29:0/3782538352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06ac146680 0x7f06ac1513b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f06a8009e50 tx=0x7f06a802bf50 comp rx=0 tx=0).stop 2024-04-25T19:44:07.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.685+0000 7f06b3dac640 1 -- 172.21.15.29:0/3782538352 shutdown_connections 2024-04-25T19:44:07.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.685+0000 7f06b3dac640 1 --2- 172.21.15.29:0/3782538352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06ac146680 0x7f06ac1513b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.685+0000 7f06b3dac640 1 --2- 172.21.15.29:0/3782538352 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f06ac0a55d0 0x7f06ac146140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.685+0000 7f06b3dac640 1 -- 172.21.15.29:0/3782538352 >> 172.21.15.29:0/3782538352 conn(0x7f06ac0ab3f0 msgr2=0x7f06ac0ab800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:07.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.685+0000 7f06b3dac640 1 -- 172.21.15.29:0/3782538352 shutdown_connections 2024-04-25T19:44:07.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.685+0000 7f06b3dac640 1 -- 172.21.15.29:0/3782538352 wait complete. 2024-04-25T19:44:07.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.687+0000 7f06b3dac640 1 Processor -- start 2024-04-25T19:44:07.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.687+0000 7f06b3dac640 1 -- start start 2024-04-25T19:44:07.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.688+0000 7f06b3dac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06ac0a55d0 0x7f06ac141820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:07.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.688+0000 7f06b3dac640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06ac0ae260 0x7f06ac140560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:07.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.688+0000 7f06b3dac640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f06ac146680 0x7f06ac13aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.688+0000 7f06b3dac640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f06ac1518f0 con 0x7f06ac0a55d0 2024-04-25T19:44:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.688+0000 7f06b3dac640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f06ac0a5bf0 con 0x7f06ac0ae260 2024-04-25T19:44:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.688+0000 7f06b3dac640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f06ac151a70 con 0x7f06ac146680 2024-04-25T19:44:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.688+0000 7f06b2349640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f06ac146680 0x7f06ac13aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.688+0000 7f06b1b48640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06ac0a55d0 0x7f06ac141820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.688+0000 7f06b1b48640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06ac0a55d0 0x7f06ac141820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36410/0 (socket says 172.21.15.29:36410) 2024-04-25T19:44:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.688+0000 7f06b1b48640 1 -- 172.21.15.29:0/2385352131 learned_addr learned my addr 172.21.15.29:0/2385352131 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.688+0000 7f06b2349640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f06ac146680 0x7f06ac13aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34308/0 (socket says 172.21.15.29:34308) 2024-04-25T19:44:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.688+0000 7f06b1347640 1 --2- 172.21.15.29:0/2385352131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06ac0ae260 0x7f06ac140560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.688+0000 7f06b2349640 1 -- 172.21.15.29:0/2385352131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06ac0ae260 msgr2=0x7f06ac140560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.688+0000 7f06b2349640 1 --2- 172.21.15.29:0/2385352131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06ac0ae260 0x7f06ac140560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.688+0000 7f06b2349640 1 -- 172.21.15.29:0/2385352131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06ac0a55d0 msgr2=0x7f06ac141820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.688+0000 7f06b2349640 1 --2- 172.21.15.29:0/2385352131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06ac0a55d0 0x7f06ac141820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:07.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.688+0000 7f06b2349640 1 -- 172.21.15.29:0/2385352131 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06ac141d60 con 0x7f06ac146680 2024-04-25T19:44:07.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.688+0000 7f06b1b48640 1 --2- 172.21.15.29:0/2385352131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06ac0a55d0 0x7f06ac141820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:07.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.688+0000 7f06b2349640 1 --2- 172.21.15.29:0/2385352131 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f06ac146680 0x7f06ac13aab0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f06a8032000 tx=0x7f06a8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:07.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.689+0000 7f06a27fc640 1 -- 172.21.15.29:0/2385352131 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a8043020 con 0x7f06ac146680 2024-04-25T19:44:07.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.689+0000 7f06a27fc640 1 -- 172.21.15.29:0/2385352131 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06a803e450 con 0x7f06ac146680 2024-04-25T19:44:07.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.689+0000 7f06b3dac640 1 -- 172.21.15.29:0/2385352131 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06ac1397a0 con 0x7f06ac146680 2024-04-25T19:44:07.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.689+0000 7f06a27fc640 1 -- 172.21.15.29:0/2385352131 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a8009510 con 0x7f06ac146680 2024-04-25T19:44:07.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.689+0000 7f06b3dac640 1 -- 172.21.15.29:0/2385352131 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f06ac140aa0 con 0x7f06ac146680 2024-04-25T19:44:07.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.690+0000 7f06b3dac640 1 -- 172.21.15.29:0/2385352131 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06ac13a7b0 con 0x7f06ac146680 2024-04-25T19:44:07.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.691+0000 7f06a27fc640 1 -- 172.21.15.29:0/2385352131 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f06a8046020 con 0x7f06ac146680 2024-04-25T19:44:07.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.691+0000 7f06a27fc640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:07.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.692+0000 7f06a27fc640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:07.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.692+0000 7f06a27fc640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:07.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.692+0000 7f06a27fc640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:07.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.692+0000 7f06a27fc640 1 --2- 172.21.15.29:0/2385352131 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f06900ac530 0x7f06900aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:07.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.692+0000 7f06a27fc640 1 -- 172.21.15.29:0/2385352131 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3482+0+0 (secure 0 0 0) 0x7f06a80fbaa0 con 0x7f06ac146680 2024-04-25T19:44:07.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.692+0000 7f06b1b48640 1 --2- 172.21.15.29:0/2385352131 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f06900ac530 0x7f06900aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:07.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.693+0000 7f06b1b48640 1 --2- 172.21.15.29:0/2385352131 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f06900ac530 0x7f06900aea40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f06a4001b90 tx=0x7f06a4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:07.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.695+0000 7f06a27fc640 1 -- 172.21.15.29:0/2385352131 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f06a804d030 con 0x7f06ac146680 2024-04-25T19:44:07.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:07.885+0000 7f06b3dac640 1 -- 172.21.15.29:0/2385352131 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f06ac0bb770 con 0x7f06ac146680 2024-04-25T19:44:08.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.019+0000 7f06a27fc640 1 -- 172.21.15.29:0/2385352131 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7f06ac0bb770 con 0x7f06ac146680 2024-04-25T19:44:08.020 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2024-04-25T19:44:08.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.021+0000 7f06b3dac640 10 mgrc shutdown 2024-04-25T19:44:08.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.021+0000 7f06b3dac640 1 -- 172.21.15.29:0/2385352131 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f06900ac530 msgr2=0x7f06900aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:08.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.021+0000 7f06b3dac640 1 --2- 172.21.15.29:0/2385352131 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f06900ac530 0x7f06900aea40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f06a4001b90 tx=0x7f06a4009040 comp rx=0 tx=0).stop 2024-04-25T19:44:08.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.021+0000 7f06b3dac640 1 -- 172.21.15.29:0/2385352131 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f06ac146680 msgr2=0x7f06ac13aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:08.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.021+0000 7f06b3dac640 1 --2- 172.21.15.29:0/2385352131 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f06ac146680 0x7f06ac13aab0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f06a8032000 tx=0x7f06a8004a40 comp rx=0 tx=0).stop 2024-04-25T19:44:08.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.021+0000 7f06b3dac640 1 -- 172.21.15.29:0/2385352131 shutdown_connections 2024-04-25T19:44:08.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.021+0000 7f06b3dac640 1 --2- 172.21.15.29:0/2385352131 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f06900ac530 0x7f06900aea40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:08.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.021+0000 7f06b3dac640 1 --2- 172.21.15.29:0/2385352131 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f06ac146680 0x7f06ac13aab0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:08.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.021+0000 7f06b3dac640 1 --2- 172.21.15.29:0/2385352131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06ac0ae260 0x7f06ac140560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:08.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.021+0000 7f06b3dac640 1 --2- 172.21.15.29:0/2385352131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06ac0a55d0 0x7f06ac141820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:08.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.021+0000 7f06b3dac640 1 -- 172.21.15.29:0/2385352131 >> 172.21.15.29:0/2385352131 conn(0x7f06ac0ab3f0 msgr2=0x7f06ac145e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:08.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.021+0000 7f06b3dac640 1 -- 172.21.15.29:0/2385352131 shutdown_connections 2024-04-25T19:44:08.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.022+0000 7f06b3dac640 1 -- 172.21.15.29:0/2385352131 wait complete. 2024-04-25T19:44:08.034 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T19:44:08.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.151+0000 7f14c66f1640 1 Processor -- start 2024-04-25T19:44:08.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.151+0000 7f14c66f1640 1 -- start start 2024-04-25T19:44:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.152+0000 7f14c66f1640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f14c00a57d0 0x7f14c0146dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.152+0000 7f14c66f1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14c00a5be0 con 0x7f14c0147300 2024-04-25T19:44:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.152+0000 7f14c66f1640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14c00a9e80 con 0x7f14c00adf50 2024-04-25T19:44:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.152+0000 7f14c66f1640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f14c00ae540 con 0x7f14c00a57d0 2024-04-25T19:44:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.152+0000 7f14c4c8e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f14c0147300 0x7f14c0151440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37966/0 (socket says 172.21.15.29:37966) 2024-04-25T19:44:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.152+0000 7f14c4c8e640 1 -- 172.21.15.29:0/790901578 learned_addr learned my addr 172.21.15.29:0/790901578 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.152+0000 7f14bf7fe640 1 --2- 172.21.15.29:0/790901578 >> v2:172.21.15.102:3300/0 conn(0x7f14c00a57d0 0x7f14c0146dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.153+0000 7f14beffd640 1 -- 172.21.15.29:0/790901578 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 304846243 0 0) 0x7f14b8001490 con 0x7f14c0147300 2024-04-25T19:44:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.153+0000 7f14beffd640 1 -- 172.21.15.29:0/790901578 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14ac003520 con 0x7f14c0147300 2024-04-25T19:44:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.153+0000 7f14beffd640 1 -- 172.21.15.29:0/790901578 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 145858293 0 0) 0x7f14c00a9e80 con 0x7f14c00adf50 2024-04-25T19:44:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.153+0000 7f14beffd640 1 -- 172.21.15.29:0/790901578 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14b8001490 con 0x7f14c00adf50 2024-04-25T19:44:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.153+0000 7f14bf7fe640 1 -- 172.21.15.29:0/790901578 >> v1:172.21.15.102:6789/0 conn(0x7f14c00adf50 legacy=0x7f14c00a9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.153+0000 7f14bf7fe640 1 -- 172.21.15.29:0/790901578 >> v1:172.21.15.29:6789/0 conn(0x7f14c0147300 legacy=0x7f14c0151440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.153+0000 7f14bf7fe640 1 -- 172.21.15.29:0/790901578 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14c0147a50 con 0x7f14c00a57d0 2024-04-25T19:44:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.154+0000 7f14bf7fe640 1 --2- 172.21.15.29:0/790901578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14c00a57d0 0x7f14c0146dc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f14b00097a0 tx=0x7f14b002b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb080069c8897bbe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.154+0000 7f14beffd640 1 -- 172.21.15.29:0/790901578 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b002f810 con 0x7f14c00a57d0 2024-04-25T19:44:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.154+0000 7f14beffd640 1 -- 172.21.15.29:0/790901578 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14b002f9b0 con 0x7f14c00a57d0 2024-04-25T19:44:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.154+0000 7f14beffd640 1 -- 172.21.15.29:0/790901578 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b002fd80 con 0x7f14c00a57d0 2024-04-25T19:44:08.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.154+0000 7f14c66f1640 1 -- 172.21.15.29:0/790901578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14c00a57d0 msgr2=0x7f14c0146dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.154+0000 7f14c66f1640 1 --2- 172.21.15.29:0/790901578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14c00a57d0 0x7f14c0146dc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f14b00097a0 tx=0x7f14b002b9f0 comp rx=0 tx=0).stop 2024-04-25T19:44:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.155+0000 7f14c66f1640 1 -- 172.21.15.29:0/790901578 shutdown_connections 2024-04-25T19:44:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.155+0000 7f14c66f1640 1 --2- 172.21.15.29:0/790901578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14c00a57d0 0x7f14c0146dc0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.155+0000 7f14c66f1640 1 -- 172.21.15.29:0/790901578 >> 172.21.15.29:0/790901578 conn(0x7f14c00bc320 msgr2=0x7f14c00bc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.155+0000 7f14c66f1640 1 -- 172.21.15.29:0/790901578 shutdown_connections 2024-04-25T19:44:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.155+0000 7f14c66f1640 1 -- 172.21.15.29:0/790901578 wait complete. 2024-04-25T19:44:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.156+0000 7f14c66f1640 1 Processor -- start 2024-04-25T19:44:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.156+0000 7f14c66f1640 1 -- start start 2024-04-25T19:44:08.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.157+0000 7f14c66f1640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14c00a57d0 0x7f14c0149b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:08.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.157+0000 7f14c66f1640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f14c00adf50 0x7f14c014b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:08.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.157+0000 7f14c66f1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14c0147300 0x7f14c014bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:08.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.157+0000 7f14c66f1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f14c0147710 con 0x7f14c0147300 2024-04-25T19:44:08.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.157+0000 7f14c66f1640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f14c00ae360 con 0x7f14c00a57d0 2024-04-25T19:44:08.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.157+0000 7f14c66f1640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f14c0152b60 con 0x7f14c00adf50 2024-04-25T19:44:08.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.157+0000 7f14c4c8e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14c0147300 0x7f14c014bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:08.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.157+0000 7f14bf7fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f14c00adf50 0x7f14c014b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:08.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.157+0000 7f14c4c8e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14c0147300 0x7f14c014bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36414/0 (socket says 172.21.15.29:36414) 2024-04-25T19:44:08.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.157+0000 7f14c4c8e640 1 -- 172.21.15.29:0/2388321982 learned_addr learned my addr 172.21.15.29:0/2388321982 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:08.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.157+0000 7f14bf7fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f14c00adf50 0x7f14c014b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34312/0 (socket says 172.21.15.29:34312) 2024-04-25T19:44:08.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.157+0000 7f14bffff640 1 --2- 172.21.15.29:0/2388321982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14c00a57d0 0x7f14c0149b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:08.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.158+0000 7f14bf7fe640 1 -- 172.21.15.29:0/2388321982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14c00a57d0 msgr2=0x7f14c0149b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:08.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.158+0000 7f14bf7fe640 1 --2- 172.21.15.29:0/2388321982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14c00a57d0 0x7f14c0149b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:08.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.158+0000 7f14bf7fe640 1 -- 172.21.15.29:0/2388321982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14c0147300 msgr2=0x7f14c014bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:08.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.158+0000 7f14bf7fe640 1 --2- 172.21.15.29:0/2388321982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14c0147300 0x7f14c014bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:08.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.158+0000 7f14bf7fe640 1 -- 172.21.15.29:0/2388321982 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14c014ade0 con 0x7f14c00adf50 2024-04-25T19:44:08.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.158+0000 7f14c4c8e640 1 --2- 172.21.15.29:0/2388321982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14c0147300 0x7f14c014bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:08.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.158+0000 7f14bf7fe640 1 --2- 172.21.15.29:0/2388321982 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f14c00adf50 0x7f14c014b630 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f14b0033000 tx=0x7f14b0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:08.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.158+0000 7f14bcff9640 1 -- 172.21.15.29:0/2388321982 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b0047070 con 0x7f14c00adf50 2024-04-25T19:44:08.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.158+0000 7f14bcff9640 1 -- 172.21.15.29:0/2388321982 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14b0037810 con 0x7f14c00adf50 2024-04-25T19:44:08.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.158+0000 7f14c66f1640 1 -- 172.21.15.29:0/2388321982 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14c014a890 con 0x7f14c00adf50 2024-04-25T19:44:08.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.158+0000 7f14bcff9640 1 -- 172.21.15.29:0/2388321982 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b0037bc0 con 0x7f14c00adf50 2024-04-25T19:44:08.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.158+0000 7f14c66f1640 1 -- 172.21.15.29:0/2388321982 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f14c013e280 con 0x7f14c00adf50 2024-04-25T19:44:08.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.159+0000 7f14c66f1640 1 -- 172.21.15.29:0/2388321982 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14c0050740 con 0x7f14c00adf50 2024-04-25T19:44:08.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.160+0000 7f14bcff9640 1 -- 172.21.15.29:0/2388321982 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f14b004a070 con 0x7f14c00adf50 2024-04-25T19:44:08.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.160+0000 7f14bcff9640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:08.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.161+0000 7f14bcff9640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:08.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.161+0000 7f14bcff9640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:08.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.161+0000 7f14bcff9640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:08.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.161+0000 7f14bcff9640 1 --2- 172.21.15.29:0/2388321982 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f14a00ac530 0x7f14a00aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:08.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.162+0000 7f14bcff9640 1 -- 172.21.15.29:0/2388321982 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 3852+0+0 (secure 0 0 0) 0x7f14b00facd0 con 0x7f14c00adf50 2024-04-25T19:44:08.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.162+0000 7f14bffff640 1 --2- 172.21.15.29:0/2388321982 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f14a00ac530 0x7f14a00aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:08.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.162+0000 7f14bffff640 1 --2- 172.21.15.29:0/2388321982 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f14a00ac530 0x7f14a00aea40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f14a80011e0 tx=0x7f14a8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:08.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.166+0000 7f14bcff9640 1 -- 172.21.15.29:0/2388321982 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f14b007eb80 con 0x7f14c00adf50 2024-04-25T19:44:08.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:08.350+0000 7f14c66f1640 1 -- 172.21.15.29:0/2388321982 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f14c00aa600 con 0x7f14c00adf50 2024-04-25T19:44:09.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.026+0000 7f14bcff9640 1 -- 172.21.15.29:0/2388321982 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) ==== 170+0+0 (secure 0 0 0) 0x7f14b0033030 con 0x7f14c00adf50 2024-04-25T19:44:09.028 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2024-04-25T19:44:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.030+0000 7f14c66f1640 10 mgrc shutdown 2024-04-25T19:44:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.030+0000 7f14c66f1640 1 -- 172.21.15.29:0/2388321982 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f14a00ac530 msgr2=0x7f14a00aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.030+0000 7f14c66f1640 1 --2- 172.21.15.29:0/2388321982 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f14a00ac530 0x7f14a00aea40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f14a80011e0 tx=0x7f14a8009040 comp rx=0 tx=0).stop 2024-04-25T19:44:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.030+0000 7f14c66f1640 1 -- 172.21.15.29:0/2388321982 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f14c00adf50 msgr2=0x7f14c014b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.030+0000 7f14c66f1640 1 --2- 172.21.15.29:0/2388321982 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f14c00adf50 0x7f14c014b630 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f14b0033000 tx=0x7f14b0004730 comp rx=0 tx=0).stop 2024-04-25T19:44:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.031+0000 7f14c66f1640 1 -- 172.21.15.29:0/2388321982 shutdown_connections 2024-04-25T19:44:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.031+0000 7f14c66f1640 1 --2- 172.21.15.29:0/2388321982 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f14a00ac530 0x7f14a00aea40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.031+0000 7f14c66f1640 1 --2- 172.21.15.29:0/2388321982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14c0147300 0x7f14c014bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.031+0000 7f14c66f1640 1 --2- 172.21.15.29:0/2388321982 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f14c00adf50 0x7f14c014b630 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.031+0000 7f14c66f1640 1 --2- 172.21.15.29:0/2388321982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14c00a57d0 0x7f14c0149b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.031+0000 7f14c66f1640 1 -- 172.21.15.29:0/2388321982 >> 172.21.15.29:0/2388321982 conn(0x7f14c00bc320 msgr2=0x7f14c00a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.031+0000 7f14c66f1640 1 -- 172.21.15.29:0/2388321982 shutdown_connections 2024-04-25T19:44:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.031+0000 7f14c66f1640 1 -- 172.21.15.29:0/2388321982 wait complete. 2024-04-25T19:44:09.048 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T19:44:09.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.167+0000 7faaae345640 1 Processor -- start 2024-04-25T19:44:09.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.168+0000 7faaae345640 1 -- start start 2024-04-25T19:44:09.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.168+0000 7faaae345640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7faaa806da90 0x7faaa80a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:09.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.169+0000 7faaae345640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaa8146c60 con 0x7faaa80ad450 2024-04-25T19:44:09.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.169+0000 7faaae345640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaa80ad9d0 con 0x7faaa81472d0 2024-04-25T19:44:09.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.169+0000 7faaae345640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7faaa80a5c10 con 0x7faaa806da90 2024-04-25T19:44:09.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.169+0000 7faaa7fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7faaa806da90 0x7faaa80a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:09.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.169+0000 7faaac8e2640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7faaa80ad450 0x7faaa81513f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37968/0 (socket says 172.21.15.29:37968) 2024-04-25T19:44:09.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.169+0000 7faaac8e2640 1 -- 172.21.15.29:0/2373131985 learned_addr learned my addr 172.21.15.29:0/2373131985 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:09.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.169+0000 7faaa6ffd640 1 -- 172.21.15.29:0/2373131985 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 813377590 0 0) 0x7faaa8146c60 con 0x7faaa80ad450 2024-04-25T19:44:09.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.169+0000 7faaa6ffd640 1 -- 172.21.15.29:0/2373131985 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa90003520 con 0x7faaa80ad450 2024-04-25T19:44:09.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.169+0000 7faaa7fff640 1 -- 172.21.15.29:0/2373131985 >> v1:172.21.15.102:6789/0 conn(0x7faaa81472d0 legacy=0x7faaa80acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:09.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.169+0000 7faaa7fff640 1 -- 172.21.15.29:0/2373131985 >> v1:172.21.15.29:6789/0 conn(0x7faaa80ad450 legacy=0x7faaa81513f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:09.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.169+0000 7faaa7fff640 1 -- 172.21.15.29:0/2373131985 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaa81476e0 con 0x7faaa806da90 2024-04-25T19:44:09.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.170+0000 7faaa7fff640 1 --2- 172.21.15.29:0/2373131985 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7faaa806da90 0x7faaa80a56d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faa9c0097d0 tx=0x7faa9c02b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a652d7d6f86820f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:09.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.170+0000 7faaa6ffd640 1 -- 172.21.15.29:0/2373131985 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9c034070 con 0x7faaa806da90 2024-04-25T19:44:09.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.170+0000 7faaa6ffd640 1 -- 172.21.15.29:0/2373131985 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa9c030880 con 0x7faaa806da90 2024-04-25T19:44:09.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.170+0000 7faaa6ffd640 1 -- 172.21.15.29:0/2373131985 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9c030c50 con 0x7faaa806da90 2024-04-25T19:44:09.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.171+0000 7faaae345640 1 -- 172.21.15.29:0/2373131985 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7faaa806da90 msgr2=0x7faaa80a56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:09.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.171+0000 7faaae345640 1 --2- 172.21.15.29:0/2373131985 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7faaa806da90 0x7faaa80a56d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faa9c0097d0 tx=0x7faa9c02b9c0 comp rx=0 tx=0).stop 2024-04-25T19:44:09.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.171+0000 7faaae345640 1 -- 172.21.15.29:0/2373131985 shutdown_connections 2024-04-25T19:44:09.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.171+0000 7faaae345640 1 --2- 172.21.15.29:0/2373131985 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7faaa806da90 0x7faaa80a56d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:09.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.171+0000 7faaae345640 1 -- 172.21.15.29:0/2373131985 >> 172.21.15.29:0/2373131985 conn(0x7faaa80bc3b0 msgr2=0x7faaa80bc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:09.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.171+0000 7faaae345640 1 -- 172.21.15.29:0/2373131985 shutdown_connections 2024-04-25T19:44:09.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.171+0000 7faaae345640 1 -- 172.21.15.29:0/2373131985 wait complete. 2024-04-25T19:44:09.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.172+0000 7faaae345640 1 Processor -- start 2024-04-25T19:44:09.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.172+0000 7faaae345640 1 -- start start 2024-04-25T19:44:09.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.173+0000 7faaae345640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7faaa806da90 0x7faaa809f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:09.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.173+0000 7faaae345640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faaa80ad450 0x7faaa809e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:09.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.173+0000 7faaae345640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaa81472d0 0x7faaa81e0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:09.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.173+0000 7faaae345640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faaa8152d20 con 0x7faaa80ad450 2024-04-25T19:44:09.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.173+0000 7faaae345640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faaa8147ab0 con 0x7faaa81472d0 2024-04-25T19:44:09.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.173+0000 7faaae345640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7faaa8152ea0 con 0x7faaa806da90 2024-04-25T19:44:09.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.173+0000 7faaa7fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7faaa806da90 0x7faaa809f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:09.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.173+0000 7faaa7fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7faaa806da90 0x7faaa809f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34324/0 (socket says 172.21.15.29:34324) 2024-04-25T19:44:09.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.173+0000 7faaa7fff640 1 -- 172.21.15.29:0/3872308006 learned_addr learned my addr 172.21.15.29:0/3872308006 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:09.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.173+0000 7faaa77fe640 1 --2- 172.21.15.29:0/3872308006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faaa80ad450 0x7faaa809e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:09.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.174+0000 7faaac8e2640 1 --2- 172.21.15.29:0/3872308006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaa81472d0 0x7faaa81e0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:09.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.174+0000 7faaa7fff640 1 -- 172.21.15.29:0/3872308006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaa81472d0 msgr2=0x7faaa81e0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:09.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.174+0000 7faaa7fff640 1 --2- 172.21.15.29:0/3872308006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaa81472d0 0x7faaa81e0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:09.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.174+0000 7faaa7fff640 1 -- 172.21.15.29:0/3872308006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faaa80ad450 msgr2=0x7faaa809e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:09.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.174+0000 7faaa7fff640 1 --2- 172.21.15.29:0/3872308006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faaa80ad450 0x7faaa809e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:09.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.174+0000 7faaa7fff640 1 -- 172.21.15.29:0/3872308006 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaa80a1700 con 0x7faaa806da90 2024-04-25T19:44:09.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.174+0000 7faaa7fff640 1 --2- 172.21.15.29:0/3872308006 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7faaa806da90 0x7faaa809f070 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7faa9c02bfd0 tx=0x7faa9c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:09.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.174+0000 7faaa4ff9640 1 -- 172.21.15.29:0/3872308006 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9c034040 con 0x7faaa806da90 2024-04-25T19:44:09.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.174+0000 7faaa4ff9640 1 -- 172.21.15.29:0/3872308006 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa9c03ada0 con 0x7faaa806da90 2024-04-25T19:44:09.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.174+0000 7faaae345640 1 -- 172.21.15.29:0/3872308006 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaa80a1e00 con 0x7faaa806da90 2024-04-25T19:44:09.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.174+0000 7faaa4ff9640 1 -- 172.21.15.29:0/3872308006 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9c044630 con 0x7faaa806da90 2024-04-25T19:44:09.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.174+0000 7faaae345640 1 -- 172.21.15.29:0/3872308006 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faaa81e1300 con 0x7faaa806da90 2024-04-25T19:44:09.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.175+0000 7faaae345640 1 -- 172.21.15.29:0/3872308006 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaa80507b0 con 0x7faaa806da90 2024-04-25T19:44:09.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.177+0000 7faaa4ff9640 1 -- 172.21.15.29:0/3872308006 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7faa9c037070 con 0x7faaa806da90 2024-04-25T19:44:09.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.177+0000 7faaa4ff9640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:09.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.178+0000 7faaa4ff9640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:09.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.178+0000 7faaa4ff9640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:09.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.178+0000 7faaa4ff9640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:09.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.178+0000 7faaa4ff9640 1 --2- 172.21.15.29:0/3872308006 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7faa940ac530 0x7faa940aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:09.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.178+0000 7faaa77fe640 1 --2- 172.21.15.29:0/3872308006 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7faa940ac530 0x7faa940aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:09.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.178+0000 7faaa4ff9640 1 -- 172.21.15.29:0/3872308006 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4234+0+0 (secure 0 0 0) 0x7faa9c0fb990 con 0x7faaa806da90 2024-04-25T19:44:09.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.178+0000 7faaa77fe640 1 --2- 172.21.15.29:0/3872308006 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7faa940ac530 0x7faa940aea40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7faaa81520a0 tx=0x7faa98009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:09.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.181+0000 7faaa4ff9640 1 -- 172.21.15.29:0/3872308006 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faa9c03f070 con 0x7faaa806da90 2024-04-25T19:44:09.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:09.408+0000 7faaae345640 1 -- 172.21.15.29:0/3872308006 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7faaa80ba340 con 0x7faaa806da90 2024-04-25T19:44:10.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.029+0000 7faaa4ff9640 1 -- 172.21.15.29:0/3872308006 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) ==== 172+0+0 (secure 0 0 0) 0x7faa9c0c6010 con 0x7faaa806da90 2024-04-25T19:44:10.030 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 3 and data pool 4 2024-04-25T19:44:10.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.033+0000 7faaae345640 10 mgrc shutdown 2024-04-25T19:44:10.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.033+0000 7faaae345640 1 -- 172.21.15.29:0/3872308006 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7faa940ac530 msgr2=0x7faa940aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:10.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.033+0000 7faaae345640 1 --2- 172.21.15.29:0/3872308006 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7faa940ac530 0x7faa940aea40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7faaa81520a0 tx=0x7faa98009040 comp rx=0 tx=0).stop 2024-04-25T19:44:10.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.033+0000 7faaae345640 1 -- 172.21.15.29:0/3872308006 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7faaa806da90 msgr2=0x7faaa809f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:10.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.033+0000 7faaae345640 1 --2- 172.21.15.29:0/3872308006 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7faaa806da90 0x7faaa809f070 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7faa9c02bfd0 tx=0x7faa9c0046c0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.033+0000 7faaae345640 1 -- 172.21.15.29:0/3872308006 shutdown_connections 2024-04-25T19:44:10.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.033+0000 7faaae345640 1 --2- 172.21.15.29:0/3872308006 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7faa940ac530 0x7faa940aea40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.033+0000 7faaae345640 1 --2- 172.21.15.29:0/3872308006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaa81472d0 0x7faaa81e0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.033+0000 7faaae345640 1 --2- 172.21.15.29:0/3872308006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faaa80ad450 0x7faaa809e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.033+0000 7faaae345640 1 --2- 172.21.15.29:0/3872308006 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7faaa806da90 0x7faaa809f070 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.033+0000 7faaae345640 1 -- 172.21.15.29:0/3872308006 >> 172.21.15.29:0/3872308006 conn(0x7faaa80bc3b0 msgr2=0x7faaa80bb030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:10.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.033+0000 7faaae345640 1 -- 172.21.15.29:0/3872308006 shutdown_connections 2024-04-25T19:44:10.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.033+0000 7faaae345640 1 -- 172.21.15.29:0/3872308006 wait complete. 2024-04-25T19:44:10.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:44:10.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.168+0000 7fb7fa312640 1 Processor -- start 2024-04-25T19:44:10.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.168+0000 7fb7fa312640 1 -- start start 2024-04-25T19:44:10.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.169+0000 7fb7fa312640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb7f4150590 0x7fb7f41529b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:10.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.169+0000 7fb7fa312640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7f4144650 con 0x7fb7f4143e30 2024-04-25T19:44:10.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.169+0000 7fb7fa312640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7f40a5a90 con 0x7fb7f4144240 2024-04-25T19:44:10.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.169+0000 7fb7fa312640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fb7f4147790 con 0x7fb7f4150590 2024-04-25T19:44:10.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.169+0000 7fb7f3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb7f4143e30 0x7fb7f40acaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37980/0 (socket says 172.21.15.29:37980) 2024-04-25T19:44:10.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.169+0000 7fb7f3fff640 1 -- 172.21.15.29:0/1646181876 learned_addr learned my addr 172.21.15.29:0/1646181876 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:10.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.169+0000 7fb7f88af640 1 --2- 172.21.15.29:0/1646181876 >> v2:172.21.15.102:3300/0 conn(0x7fb7f4150590 0x7fb7f41529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:10.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.169+0000 7fb7f2ffd640 1 -- 172.21.15.29:0/1646181876 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3498160186 0 0) 0x7fb7e4001110 con 0x7fb7f4143e30 2024-04-25T19:44:10.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.169+0000 7fb7f2ffd640 1 -- 172.21.15.29:0/1646181876 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7dc003520 con 0x7fb7f4143e30 2024-04-25T19:44:10.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.169+0000 7fb7f2ffd640 1 -- 172.21.15.29:0/1646181876 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1978635632 0 0) 0x7fb7f40a5a90 con 0x7fb7f4144240 2024-04-25T19:44:10.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.169+0000 7fb7f2ffd640 1 -- 172.21.15.29:0/1646181876 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7e4001110 con 0x7fb7f4144240 2024-04-25T19:44:10.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.170+0000 7fb7f2ffd640 1 -- 172.21.15.29:0/1646181876 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2477732959 0 0) 0x7fb7dc003520 con 0x7fb7f4143e30 2024-04-25T19:44:10.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.170+0000 7fb7f88af640 1 -- 172.21.15.29:0/1646181876 >> v1:172.21.15.102:6789/0 conn(0x7fb7f4144240 legacy=0x7fb7f4146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:10.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.170+0000 7fb7f88af640 1 -- 172.21.15.29:0/1646181876 >> v1:172.21.15.29:6789/0 conn(0x7fb7f4143e30 legacy=0x7fb7f40acaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:10.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.170+0000 7fb7f88af640 1 -- 172.21.15.29:0/1646181876 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f4147a50 con 0x7fb7f4150590 2024-04-25T19:44:10.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.170+0000 7fb7f88af640 1 --2- 172.21.15.29:0/1646181876 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7f4150590 0x7fb7f41529b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec009d90 tx=0x7fb7ec02bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c6476287d82362a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:10.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.171+0000 7fb7f2ffd640 1 -- 172.21.15.29:0/1646181876 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec039070 con 0x7fb7f4150590 2024-04-25T19:44:10.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.171+0000 7fb7f2ffd640 1 -- 172.21.15.29:0/1646181876 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7ec02c8b0 con 0x7fb7f4150590 2024-04-25T19:44:10.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.171+0000 7fb7f2ffd640 1 -- 172.21.15.29:0/1646181876 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e4001840 con 0x7fb7f4150590 2024-04-25T19:44:10.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.171+0000 7fb7fa312640 1 -- 172.21.15.29:0/1646181876 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7f4150590 msgr2=0x7fb7f41529b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:10.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.171+0000 7fb7fa312640 1 --2- 172.21.15.29:0/1646181876 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7f4150590 0x7fb7f41529b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec009d90 tx=0x7fb7ec02bd80 comp rx=0 tx=0).stop 2024-04-25T19:44:10.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.171+0000 7fb7fa312640 1 -- 172.21.15.29:0/1646181876 shutdown_connections 2024-04-25T19:44:10.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.171+0000 7fb7fa312640 1 --2- 172.21.15.29:0/1646181876 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7f4150590 0x7fb7f41529b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.171+0000 7fb7fa312640 1 -- 172.21.15.29:0/1646181876 >> 172.21.15.29:0/1646181876 conn(0x7fb7f40aafc0 msgr2=0x7fb7f40ab3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:10.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.171+0000 7fb7fa312640 1 -- 172.21.15.29:0/1646181876 shutdown_connections 2024-04-25T19:44:10.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.171+0000 7fb7fa312640 1 -- 172.21.15.29:0/1646181876 wait complete. 2024-04-25T19:44:10.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.173+0000 7fb7fa312640 1 Processor -- start 2024-04-25T19:44:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.173+0000 7fb7fa312640 1 -- start start 2024-04-25T19:44:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.174+0000 7fb7fa312640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7f4143e30 0x7fb7f40a7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.174+0000 7fb7fa312640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb7f4144240 0x7fb7f4147fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.174+0000 7fb7fa312640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7f4150590 0x7fb7f40a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.174+0000 7fb7fa312640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f41474a0 con 0x7fb7f4150590 2024-04-25T19:44:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.174+0000 7fb7fa312640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f4147320 con 0x7fb7f4143e30 2024-04-25T19:44:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.174+0000 7fb7fa312640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fb7f4152ef0 con 0x7fb7f4144240 2024-04-25T19:44:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.174+0000 7fb7f88af640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7f4150590 0x7fb7f40a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.174+0000 7fb7f37fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb7f4144240 0x7fb7f4147fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.174+0000 7fb7f88af640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7f4150590 0x7fb7f40a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36424/0 (socket says 172.21.15.29:36424) 2024-04-25T19:44:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.174+0000 7fb7f88af640 1 -- 172.21.15.29:0/3210371430 learned_addr learned my addr 172.21.15.29:0/3210371430 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.174+0000 7fb7f3fff640 1 --2- 172.21.15.29:0/3210371430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7f4143e30 0x7fb7f40a7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.174+0000 7fb7f88af640 1 -- 172.21.15.29:0/3210371430 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb7f4144240 msgr2=0x7fb7f4147fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.174+0000 7fb7f88af640 1 --2- 172.21.15.29:0/3210371430 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb7f4144240 0x7fb7f4147fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.174+0000 7fb7f88af640 1 -- 172.21.15.29:0/3210371430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7f4143e30 msgr2=0x7fb7f40a7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.174+0000 7fb7f88af640 1 --2- 172.21.15.29:0/3210371430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7f4143e30 0x7fb7f40a7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.174+0000 7fb7f88af640 1 -- 172.21.15.29:0/3210371430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f40a7d50 con 0x7fb7f4150590 2024-04-25T19:44:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.175+0000 7fb7f88af640 1 --2- 172.21.15.29:0/3210371430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7f4150590 0x7fb7f40a46c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec037040 tx=0x7fb7ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.175+0000 7fb7f0ff9640 1 -- 172.21.15.29:0/3210371430 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec039050 con 0x7fb7f4150590 2024-04-25T19:44:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.175+0000 7fb7f0ff9640 1 -- 172.21.15.29:0/3210371430 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7ec03a040 con 0x7fb7f4150590 2024-04-25T19:44:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.175+0000 7fb7fa312640 1 -- 172.21.15.29:0/3210371430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7f4148520 con 0x7fb7f4150590 2024-04-25T19:44:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.175+0000 7fb7f0ff9640 1 -- 172.21.15.29:0/3210371430 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec009450 con 0x7fb7f4150590 2024-04-25T19:44:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.175+0000 7fb7fa312640 1 -- 172.21.15.29:0/3210371430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7f4150e20 con 0x7fb7f4150590 2024-04-25T19:44:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.176+0000 7fb7fa312640 1 -- 172.21.15.29:0/3210371430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7f41518d0 con 0x7fb7f4150590 2024-04-25T19:44:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.177+0000 7fb7f0ff9640 1 -- 172.21.15.29:0/3210371430 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7fb7ec047020 con 0x7fb7f4150590 2024-04-25T19:44:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.177+0000 7fb7f0ff9640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.178+0000 7fb7f0ff9640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.178+0000 7fb7f0ff9640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:10.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.178+0000 7fb7f0ff9640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:10.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.178+0000 7fb7f0ff9640 1 --2- 172.21.15.29:0/3210371430 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fb7e00ac530 0x7fb7e00aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:10.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.178+0000 7fb7f0ff9640 1 -- 172.21.15.29:0/3210371430 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4334+0+0 (secure 0 0 0) 0x7fb7ec0fb9f0 con 0x7fb7f4150590 2024-04-25T19:44:10.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.178+0000 7fb7f3fff640 1 --2- 172.21.15.29:0/3210371430 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fb7e00ac530 0x7fb7e00aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:10.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.179+0000 7fb7f3fff640 1 --2- 172.21.15.29:0/3210371430 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fb7e00ac530 0x7fb7e00aea40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb7e40013b0 tx=0x7fb7e4001dd0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:10.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.182+0000 7fb7f0ff9640 1 -- 172.21.15.29:0/3210371430 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb7f41518d0 con 0x7fb7f4150590 2024-04-25T19:44:10.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.369+0000 7fb7fa312640 1 -- 172.21.15.29:0/3210371430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb7f40507b0 con 0x7fb7f4150590 2024-04-25T19:44:10.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.370+0000 7fb7f0ff9640 1 -- 172.21.15.29:0/3210371430 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+13525 (secure 0 0 0) 0x7fb7ec04e020 con 0x7fb7f4150590 2024-04-25T19:44:10.372 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:10.372 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":19,"fsid":"1b3c39b5-e787-409e-b0bd-f4f491c89d72","created":"2024-04-25T19:43:47.854697+0000","modified":"2024-04-25T19:44:09.410606+0000","last_up_change":"2024-04-25T19:43:57.853237+0000","last_in_change":"2024-04-25T19:43:49.022776+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:43:58.096700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T19:44:03.617011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:44:07.890493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2024-04-25T19:44:08.355570+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d805223-ee84-494f-bd31-59091906aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6801","nonce":695704147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6803","nonce":695704147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6807","nonce":695704147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6805","nonce":695704147}]},"public_addr":"172.21.15.29:6801/695704147","cluster_addr":"172.21.15.29:6803/695704147","heartbeat_back_addr":"172.21.15.29:6807/695704147","heartbeat_front_addr":"172.21.15.29:6805/695704147","state":["exists","up"]},{"osd":1,"uuid":"607c5038-38af-440f-8371-e71758549920","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6814","nonce":3135821452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6817","nonce":3135821452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6823","nonce":3135821452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6821","nonce":3135821452}]},"public_addr":"172.21.15.102:6814/3135821452","cluster_addr":"172.21.15.102:6817/3135821452","heartbeat_back_addr":"172.21.15.102:6823/3135821452","heartbeat_front_addr":"172.21.15.102:6821/3135821452","state":["exists","up"]},{"osd":2,"uuid":"6859ae76-54bb-4c55-b0dc-5e2735c0d31c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6801","nonce":2071844144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6803","nonce":2071844144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6811","nonce":2071844144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6806","nonce":2071844144}]},"public_addr":"172.21.15.102:6801/2071844144","cluster_addr":"172.21.15.102:6803/2071844144","heartbeat_back_addr":"172.21.15.102:6811/2071844144","heartbeat_front_addr":"172.21.15.102:6806/2071844144","state":["exists","up"]},{"osd":3,"uuid":"a98b7b88-5206-4f22-84a6-78212e0d5f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6805","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6807","nonce":339944984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6810","nonce":339944984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6820","nonce":339944984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6815","nonce":339944984}]},"public_addr":"172.21.15.102:6807/339944984","cluster_addr":"172.21.15.102:6810/339944984","heartbeat_back_addr":"172.21.15.102:6820/339944984","heartbeat_front_addr":"172.21.15.102:6815/339944984","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.121529+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.201155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.264387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.198713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:44:10.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.372+0000 7fb7fa312640 10 mgrc shutdown 2024-04-25T19:44:10.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.372+0000 7fb7fa312640 1 -- 172.21.15.29:0/3210371430 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fb7e00ac530 msgr2=0x7fb7e00aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:10.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.372+0000 7fb7fa312640 1 --2- 172.21.15.29:0/3210371430 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fb7e00ac530 0x7fb7e00aea40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb7e40013b0 tx=0x7fb7e4001dd0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.372+0000 7fb7fa312640 1 -- 172.21.15.29:0/3210371430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7f4150590 msgr2=0x7fb7f40a46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:10.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.372+0000 7fb7fa312640 1 --2- 172.21.15.29:0/3210371430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7f4150590 0x7fb7f40a46c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec037040 tx=0x7fb7ec004a40 comp rx=0 tx=0).stop 2024-04-25T19:44:10.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.372+0000 7fb7fa312640 1 -- 172.21.15.29:0/3210371430 shutdown_connections 2024-04-25T19:44:10.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.372+0000 7fb7fa312640 1 --2- 172.21.15.29:0/3210371430 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fb7e00ac530 0x7fb7e00aea40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.372+0000 7fb7fa312640 1 --2- 172.21.15.29:0/3210371430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7f4150590 0x7fb7f40a46c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.372+0000 7fb7fa312640 1 --2- 172.21.15.29:0/3210371430 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb7f4144240 0x7fb7f4147fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.372+0000 7fb7fa312640 1 --2- 172.21.15.29:0/3210371430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7f4143e30 0x7fb7f40a7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.372+0000 7fb7fa312640 1 -- 172.21.15.29:0/3210371430 >> 172.21.15.29:0/3210371430 conn(0x7fb7f40aafc0 msgr2=0x7fb7f4146a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:10.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.373+0000 7fb7fa312640 1 -- 172.21.15.29:0/3210371430 shutdown_connections 2024-04-25T19:44:10.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.373+0000 7fb7fa312640 1 -- 172.21.15.29:0/3210371430 wait complete. 2024-04-25T19:44:10.385 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:44:10.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.494+0000 7f4d58c1e640 1 Processor -- start 2024-04-25T19:44:10.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.494+0000 7f4d58c1e640 1 -- start start 2024-04-25T19:44:10.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.495+0000 7f4d58c1e640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d5406da90 0x7f4d540a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:10.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.495+0000 7f4d58c1e640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4d541472d0 0x7f4d541476e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:10.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.495+0000 7f4d58c1e640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d54146c60 con 0x7f4d540acd40 2024-04-25T19:44:10.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.495+0000 7f4d58c1e640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f4d540a5c10 con 0x7f4d541472d0 2024-04-25T19:44:10.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.495+0000 7f4d58c1e640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f4d540ad9d0 con 0x7f4d5406da90 2024-04-25T19:44:10.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.495+0000 7f4d5259c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d5406da90 0x7f4d540a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:10.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.495+0000 7f4d5259c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d5406da90 0x7f4d540a56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34336/0 (socket says 172.21.15.29:34336) 2024-04-25T19:44:10.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.495+0000 7f4d5259c640 1 -- 172.21.15.29:0/27118002 learned_addr learned my addr 172.21.15.29:0/27118002 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:10.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.495+0000 7f4d5259c640 1 -- 172.21.15.29:0/27118002 >> v2:172.21.15.102:3300/0 conn(0x7f4d541472d0 msgr2=0x7f4d541476e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:10.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.495+0000 7f4d4bfff640 1 --2- 172.21.15.29:0/27118002 >> v2:172.21.15.102:3300/0 conn(0x7f4d541472d0 0x7f4d541476e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:10.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.495+0000 7f4d5259c640 1 --2- 172.21.15.29:0/27118002 >> v2:172.21.15.102:3300/0 conn(0x7f4d541472d0 0x7f4d541476e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.495+0000 7f4d5259c640 1 -- 172.21.15.29:0/27118002 >> v1:172.21.15.102:6789/0 conn(0x7f4d540acd40 legacy=0x7f4d540ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:10.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.495+0000 7f4d5259c640 1 -- 172.21.15.29:0/27118002 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d54152400 con 0x7f4d5406da90 2024-04-25T19:44:10.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.496+0000 7f4d5259c640 1 --2- 172.21.15.29:0/27118002 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d5406da90 0x7f4d540a56d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4d44009870 tx=0x7f4d4402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b50d0345fe7bc68 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:10.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.496+0000 7f4d51d9b640 1 -- 172.21.15.29:0/27118002 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d44035020 con 0x7f4d5406da90 2024-04-25T19:44:10.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.496+0000 7f4d51d9b640 1 -- 172.21.15.29:0/27118002 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d44031640 con 0x7f4d5406da90 2024-04-25T19:44:10.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.496+0000 7f4d51d9b640 1 -- 172.21.15.29:0/27118002 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4402d040 con 0x7f4d5406da90 2024-04-25T19:44:10.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.497+0000 7f4d58c1e640 1 -- 172.21.15.29:0/27118002 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d5406da90 msgr2=0x7f4d540a56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:10.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.497+0000 7f4d58c1e640 1 --2- 172.21.15.29:0/27118002 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d5406da90 0x7f4d540a56d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4d44009870 tx=0x7f4d4402b8f0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.497+0000 7f4d58c1e640 1 -- 172.21.15.29:0/27118002 shutdown_connections 2024-04-25T19:44:10.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.497+0000 7f4d58c1e640 1 --2- 172.21.15.29:0/27118002 >> v2:172.21.15.102:3300/0 conn(0x7f4d541472d0 0x7f4d541476e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.497+0000 7f4d58c1e640 1 --2- 172.21.15.29:0/27118002 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d5406da90 0x7f4d540a56d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.497+0000 7f4d58c1e640 1 -- 172.21.15.29:0/27118002 >> 172.21.15.29:0/27118002 conn(0x7f4d540bc3b0 msgr2=0x7f4d540bc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:10.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.497+0000 7f4d58c1e640 1 -- 172.21.15.29:0/27118002 shutdown_connections 2024-04-25T19:44:10.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.497+0000 7f4d58c1e640 1 -- 172.21.15.29:0/27118002 wait complete. 2024-04-25T19:44:10.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.498+0000 7f4d58c1e640 1 Processor -- start 2024-04-25T19:44:10.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.499+0000 7f4d58c1e640 1 -- start start 2024-04-25T19:44:10.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.499+0000 7f4d58c1e640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d5406da90 0x7f4d540a2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:10.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.499+0000 7f4d58c1e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d540acd40 0x7f4d540ba0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:10.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.499+0000 7f4d58c1e640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d541472d0 0x7f4d5413b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:10.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.499+0000 7f4d58c1e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4d541517c0 con 0x7f4d540acd40 2024-04-25T19:44:10.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.499+0000 7f4d58c1e640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4d54151640 con 0x7f4d5406da90 2024-04-25T19:44:10.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.499+0000 7f4d58c1e640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f4d54151940 con 0x7f4d541472d0 2024-04-25T19:44:10.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.500+0000 7f4d4bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d540acd40 0x7f4d540ba0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:10.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.500+0000 7f4d52d9d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d541472d0 0x7f4d5413b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:10.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.500+0000 7f4d52d9d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d541472d0 0x7f4d5413b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34346/0 (socket says 172.21.15.29:34346) 2024-04-25T19:44:10.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.500+0000 7f4d52d9d640 1 -- 172.21.15.29:0/128894218 learned_addr learned my addr 172.21.15.29:0/128894218 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:10.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.500+0000 7f4d4bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d540acd40 0x7f4d540ba0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36438/0 (socket says 172.21.15.29:36438) 2024-04-25T19:44:10.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.500+0000 7f4d5259c640 1 --2- 172.21.15.29:0/128894218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d5406da90 0x7f4d540a2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:10.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.500+0000 7f4d4bfff640 1 -- 172.21.15.29:0/128894218 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d541472d0 msgr2=0x7f4d5413b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:10.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.500+0000 7f4d4bfff640 1 --2- 172.21.15.29:0/128894218 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d541472d0 0x7f4d5413b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.500+0000 7f4d4bfff640 1 -- 172.21.15.29:0/128894218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d5406da90 msgr2=0x7f4d540a2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:10.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.500+0000 7f4d4bfff640 1 --2- 172.21.15.29:0/128894218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d5406da90 0x7f4d540a2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.500+0000 7f4d4bfff640 1 -- 172.21.15.29:0/128894218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d540a7a70 con 0x7f4d540acd40 2024-04-25T19:44:10.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.500+0000 7f4d52d9d640 1 --2- 172.21.15.29:0/128894218 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d541472d0 0x7f4d5413b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:10.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.500+0000 7f4d4bfff640 1 --2- 172.21.15.29:0/128894218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d540acd40 0x7f4d540ba0a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c00c5f0 tx=0x7f4d3c00cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:10.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.500+0000 7f4d4affd640 1 -- 172.21.15.29:0/128894218 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c015070 con 0x7f4d540acd40 2024-04-25T19:44:10.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.500+0000 7f4d4affd640 1 -- 172.21.15.29:0/128894218 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d3c011800 con 0x7f4d540acd40 2024-04-25T19:44:10.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.500+0000 7f4d58c1e640 1 -- 172.21.15.29:0/128894218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d5413d7d0 con 0x7f4d540acd40 2024-04-25T19:44:10.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.500+0000 7f4d4affd640 1 -- 172.21.15.29:0/128894218 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c011bd0 con 0x7f4d540acd40 2024-04-25T19:44:10.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.500+0000 7f4d58c1e640 1 -- 172.21.15.29:0/128894218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d540bae60 con 0x7f4d540acd40 2024-04-25T19:44:10.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.501+0000 7f4d58c1e640 1 -- 172.21.15.29:0/128894218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d540507b0 con 0x7f4d540acd40 2024-04-25T19:44:10.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.503+0000 7f4d4affd640 1 -- 172.21.15.29:0/128894218 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f4d3c018070 con 0x7f4d540acd40 2024-04-25T19:44:10.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.503+0000 7f4d4affd640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:10.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.503+0000 7f4d4affd640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:10.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.503+0000 7f4d4affd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:10.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.503+0000 7f4d4affd640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:10.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.503+0000 7f4d4affd640 1 --2- 172.21.15.29:0/128894218 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f4d340ac530 0x7f4d340aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:10.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.504+0000 7f4d4affd640 1 -- 172.21.15.29:0/128894218 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4334+0+0 (secure 0 0 0) 0x7f4d3c0d9460 con 0x7f4d540acd40 2024-04-25T19:44:10.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.504+0000 7f4d5259c640 1 --2- 172.21.15.29:0/128894218 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f4d340ac530 0x7f4d340aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:10.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.504+0000 7f4d5259c640 1 --2- 172.21.15.29:0/128894218 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f4d340ac530 0x7f4d340aea40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4d44036000 tx=0x7f4d4403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:10.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.507+0000 7f4d4affd640 1 -- 172.21.15.29:0/128894218 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4d3c0a39e0 con 0x7f4d540acd40 2024-04-25T19:44:10.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.690+0000 7f4d58c1e640 1 -- 172.21.15.29:0/128894218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4d5406e680 con 0x7f4d540acd40 2024-04-25T19:44:10.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.692+0000 7f4d4affd640 1 -- 172.21.15.29:0/128894218 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+13525 (secure 0 0 0) 0x7f4d3c0a7ce0 con 0x7f4d540acd40 2024-04-25T19:44:10.693 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:10.693 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":19,"fsid":"1b3c39b5-e787-409e-b0bd-f4f491c89d72","created":"2024-04-25T19:43:47.854697+0000","modified":"2024-04-25T19:44:09.410606+0000","last_up_change":"2024-04-25T19:43:57.853237+0000","last_in_change":"2024-04-25T19:43:49.022776+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:43:58.096700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T19:44:03.617011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:44:07.890493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2024-04-25T19:44:08.355570+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d805223-ee84-494f-bd31-59091906aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6801","nonce":695704147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6803","nonce":695704147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6807","nonce":695704147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6805","nonce":695704147}]},"public_addr":"172.21.15.29:6801/695704147","cluster_addr":"172.21.15.29:6803/695704147","heartbeat_back_addr":"172.21.15.29:6807/695704147","heartbeat_front_addr":"172.21.15.29:6805/695704147","state":["exists","up"]},{"osd":1,"uuid":"607c5038-38af-440f-8371-e71758549920","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6814","nonce":3135821452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6817","nonce":3135821452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6823","nonce":3135821452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6821","nonce":3135821452}]},"public_addr":"172.21.15.102:6814/3135821452","cluster_addr":"172.21.15.102:6817/3135821452","heartbeat_back_addr":"172.21.15.102:6823/3135821452","heartbeat_front_addr":"172.21.15.102:6821/3135821452","state":["exists","up"]},{"osd":2,"uuid":"6859ae76-54bb-4c55-b0dc-5e2735c0d31c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6801","nonce":2071844144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6803","nonce":2071844144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6811","nonce":2071844144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6806","nonce":2071844144}]},"public_addr":"172.21.15.102:6801/2071844144","cluster_addr":"172.21.15.102:6803/2071844144","heartbeat_back_addr":"172.21.15.102:6811/2071844144","heartbeat_front_addr":"172.21.15.102:6806/2071844144","state":["exists","up"]},{"osd":3,"uuid":"a98b7b88-5206-4f22-84a6-78212e0d5f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6805","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6807","nonce":339944984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6810","nonce":339944984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6820","nonce":339944984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6815","nonce":339944984}]},"public_addr":"172.21.15.102:6807/339944984","cluster_addr":"172.21.15.102:6810/339944984","heartbeat_back_addr":"172.21.15.102:6820/339944984","heartbeat_front_addr":"172.21.15.102:6815/339944984","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.121529+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.201155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.264387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.198713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:44:10.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.693+0000 7f4d58c1e640 10 mgrc shutdown 2024-04-25T19:44:10.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.693+0000 7f4d58c1e640 1 -- 172.21.15.29:0/128894218 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f4d340ac530 msgr2=0x7f4d340aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:10.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.693+0000 7f4d58c1e640 1 --2- 172.21.15.29:0/128894218 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f4d340ac530 0x7f4d340aea40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4d44036000 tx=0x7f4d4403c040 comp rx=0 tx=0).stop 2024-04-25T19:44:10.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.693+0000 7f4d58c1e640 1 -- 172.21.15.29:0/128894218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d540acd40 msgr2=0x7f4d540ba0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:10.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.693+0000 7f4d58c1e640 1 --2- 172.21.15.29:0/128894218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d540acd40 0x7f4d540ba0a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c00c5f0 tx=0x7f4d3c00cab0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.694+0000 7f4d58c1e640 1 -- 172.21.15.29:0/128894218 shutdown_connections 2024-04-25T19:44:10.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.694+0000 7f4d58c1e640 1 --2- 172.21.15.29:0/128894218 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f4d340ac530 0x7f4d340aea40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.694+0000 7f4d58c1e640 1 --2- 172.21.15.29:0/128894218 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d541472d0 0x7f4d5413b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.694+0000 7f4d58c1e640 1 --2- 172.21.15.29:0/128894218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d540acd40 0x7f4d540ba0a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.694+0000 7f4d58c1e640 1 --2- 172.21.15.29:0/128894218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d5406da90 0x7f4d540a2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.694+0000 7f4d58c1e640 1 -- 172.21.15.29:0/128894218 >> 172.21.15.29:0/128894218 conn(0x7f4d540bc3b0 msgr2=0x7f4d540a9630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:10.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.694+0000 7f4d58c1e640 1 -- 172.21.15.29:0/128894218 shutdown_connections 2024-04-25T19:44:10.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.694+0000 7f4d58c1e640 1 -- 172.21.15.29:0/128894218 wait complete. 2024-04-25T19:44:10.706 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T19:44:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.818+0000 7f9d171ec640 1 Processor -- start 2024-04-25T19:44:10.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.818+0000 7f9d171ec640 1 -- start start 2024-04-25T19:44:10.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.819+0000 7f9d171ec640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9d10143e30 0x7f9d10144240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:10.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.819+0000 7f9d171ec640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d100a5a90 con 0x7f9d100acaa0 2024-04-25T19:44:10.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.819+0000 7f9d171ec640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d10147790 con 0x7f9d100aceb0 2024-04-25T19:44:10.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.819+0000 7f9d171ec640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f9d10147a50 con 0x7f9d10143e30 2024-04-25T19:44:10.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.819+0000 7f9d0ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9d100acaa0 0x7f9d10146c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37988/0 (socket says 172.21.15.29:37988) 2024-04-25T19:44:10.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.819+0000 7f9d0ffff640 1 -- 172.21.15.29:0/1605742208 learned_addr learned my addr 172.21.15.29:0/1605742208 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:10.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.819+0000 7f9d14f88640 1 --2- 172.21.15.29:0/1605742208 >> v2:172.21.15.102:3300/0 conn(0x7f9d10143e30 0x7f9d10144240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:10.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.819+0000 7f9d0f7fe640 1 -- 172.21.15.29:0/1605742208 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1327993614 0 0) 0x7f9d100a5a90 con 0x7f9d100acaa0 2024-04-25T19:44:10.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.820+0000 7f9d0f7fe640 1 -- 172.21.15.29:0/1605742208 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cf8003520 con 0x7f9d100acaa0 2024-04-25T19:44:10.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.820+0000 7f9d14f88640 1 -- 172.21.15.29:0/1605742208 >> v1:172.21.15.102:6789/0 conn(0x7f9d100aceb0 legacy=0x7f9d10151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:10.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.820+0000 7f9d14f88640 1 -- 172.21.15.29:0/1605742208 >> v1:172.21.15.29:6789/0 conn(0x7f9d100acaa0 legacy=0x7f9d10146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:10.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.820+0000 7f9d0f7fe640 1 -- 172.21.15.29:0/1605742208 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1620172328 0 0) 0x7f9d10147790 con 0x7f9d100aceb0 2024-04-25T19:44:10.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.820+0000 7f9d14f88640 1 -- 172.21.15.29:0/1605742208 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d100ad2c0 con 0x7f9d10143e30 2024-04-25T19:44:10.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.821+0000 7f9d14f88640 1 --2- 172.21.15.29:0/1605742208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d10143e30 0x7f9d10144240 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9d000097a0 tx=0x7f9d0002b6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f62d129e39924bf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:10.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.821+0000 7f9d0f7fe640 1 -- 172.21.15.29:0/1605742208 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d00037070 con 0x7f9d10143e30 2024-04-25T19:44:10.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.821+0000 7f9d0f7fe640 1 -- 172.21.15.29:0/1605742208 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d00030510 con 0x7f9d10143e30 2024-04-25T19:44:10.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.821+0000 7f9d0f7fe640 1 -- 172.21.15.29:0/1605742208 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d04001670 con 0x7f9d10143e30 2024-04-25T19:44:10.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.821+0000 7f9d171ec640 1 -- 172.21.15.29:0/1605742208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d10143e30 msgr2=0x7f9d10144240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:10.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.821+0000 7f9d171ec640 1 --2- 172.21.15.29:0/1605742208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d10143e30 0x7f9d10144240 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9d000097a0 tx=0x7f9d0002b6b0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.821+0000 7f9d171ec640 1 -- 172.21.15.29:0/1605742208 shutdown_connections 2024-04-25T19:44:10.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.821+0000 7f9d171ec640 1 --2- 172.21.15.29:0/1605742208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d10143e30 0x7f9d10144240 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.821+0000 7f9d171ec640 1 -- 172.21.15.29:0/1605742208 >> 172.21.15.29:0/1605742208 conn(0x7f9d100aafc0 msgr2=0x7f9d100ab3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:10.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.822+0000 7f9d171ec640 1 -- 172.21.15.29:0/1605742208 shutdown_connections 2024-04-25T19:44:10.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.822+0000 7f9d171ec640 1 -- 172.21.15.29:0/1605742208 wait complete. 2024-04-25T19:44:10.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.823+0000 7f9d171ec640 1 Processor -- start 2024-04-25T19:44:10.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.823+0000 7f9d171ec640 1 -- start start 2024-04-25T19:44:10.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.824+0000 7f9d171ec640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9d100acaa0 0x7f9d1014e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:10.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.824+0000 7f9d171ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d100aceb0 0x7f9d1014a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:10.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.824+0000 7f9d171ec640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d10143e30 0x7f9d101e0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.824+0000 7f9d171ec640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9d10152e30 con 0x7f9d100aceb0 2024-04-25T19:44:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.824+0000 7f9d171ec640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9d10152cb0 con 0x7f9d10143e30 2024-04-25T19:44:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.824+0000 7f9d171ec640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f9d10152fb0 con 0x7f9d100acaa0 2024-04-25T19:44:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.824+0000 7f9d0ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d100aceb0 0x7f9d1014a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.824+0000 7f9d14f88640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9d100acaa0 0x7f9d1014e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.824+0000 7f9d0ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d100aceb0 0x7f9d1014a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36440/0 (socket says 172.21.15.29:36440) 2024-04-25T19:44:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.824+0000 7f9d0ffff640 1 -- 172.21.15.29:0/3993309311 learned_addr learned my addr 172.21.15.29:0/3993309311 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.824+0000 7f9d15789640 1 --2- 172.21.15.29:0/3993309311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d10143e30 0x7f9d101e0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.824+0000 7f9d0ffff640 1 -- 172.21.15.29:0/3993309311 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9d100acaa0 msgr2=0x7f9d1014e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.824+0000 7f9d0ffff640 1 --2- 172.21.15.29:0/3993309311 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9d100acaa0 0x7f9d1014e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.825+0000 7f9d0ffff640 1 -- 172.21.15.29:0/3993309311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d10143e30 msgr2=0x7f9d101e0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.825+0000 7f9d0ffff640 1 --2- 172.21.15.29:0/3993309311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d10143e30 0x7f9d101e0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.825+0000 7f9d0ffff640 1 -- 172.21.15.29:0/3993309311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d10149c00 con 0x7f9d100aceb0 2024-04-25T19:44:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.825+0000 7f9d14f88640 1 --2- 172.21.15.29:0/3993309311 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9d100acaa0 0x7f9d1014e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:10.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.825+0000 7f9d0ffff640 1 --2- 172.21.15.29:0/3993309311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d100aceb0 0x7f9d1014a320 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9d0400c820 tx=0x7f9d0400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:10.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.825+0000 7f9d0d7fa640 1 -- 172.21.15.29:0/3993309311 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d04018070 con 0x7f9d100aceb0 2024-04-25T19:44:10.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.825+0000 7f9d0d7fa640 1 -- 172.21.15.29:0/3993309311 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d04007500 con 0x7f9d100aceb0 2024-04-25T19:44:10.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.825+0000 7f9d171ec640 1 -- 172.21.15.29:0/3993309311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d10152320 con 0x7f9d100aceb0 2024-04-25T19:44:10.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.825+0000 7f9d0d7fa640 1 -- 172.21.15.29:0/3993309311 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d04004a20 con 0x7f9d100aceb0 2024-04-25T19:44:10.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.825+0000 7f9d171ec640 1 -- 172.21.15.29:0/3993309311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d1014ef50 con 0x7f9d100aceb0 2024-04-25T19:44:10.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.826+0000 7f9d171ec640 1 -- 172.21.15.29:0/3993309311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d100507b0 con 0x7f9d100aceb0 2024-04-25T19:44:10.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.828+0000 7f9d0d7fa640 1 -- 172.21.15.29:0/3993309311 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f9d0401b070 con 0x7f9d100aceb0 2024-04-25T19:44:10.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.828+0000 7f9d0d7fa640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:10.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.828+0000 7f9d0d7fa640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:10.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.828+0000 7f9d0d7fa640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:10.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.828+0000 7f9d0d7fa640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:10.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.828+0000 7f9d0d7fa640 1 --2- 172.21.15.29:0/3993309311 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f9cfc0ac560 0x7f9cfc0aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:10.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.828+0000 7f9d0d7fa640 1 -- 172.21.15.29:0/3993309311 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4334+0+0 (secure 0 0 0) 0x7f9d040d8b50 con 0x7f9d100aceb0 2024-04-25T19:44:10.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.828+0000 7f9d14f88640 1 --2- 172.21.15.29:0/3993309311 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f9cfc0ac560 0x7f9cfc0aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:10.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.829+0000 7f9d14f88640 1 --2- 172.21.15.29:0/3993309311 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f9cfc0ac560 0x7f9cfc0aea70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9d00034000 tx=0x7f9d00008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:10.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:10.831+0000 7f9d0d7fa640 1 -- 172.21.15.29:0/3993309311 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9d040a3170 con 0x7f9d100aceb0 2024-04-25T19:44:11.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:11.062+0000 7f9d171ec640 1 -- 172.21.15.29:0/3993309311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f9d100a5d30 con 0x7f9d100aceb0 2024-04-25T19:44:12.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.044+0000 7f9d0d7fa640 1 -- 172.21.15.29:0/3993309311 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7f9d040a7470 con 0x7f9d100aceb0 2024-04-25T19:44:12.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.047+0000 7f9d171ec640 10 mgrc shutdown 2024-04-25T19:44:12.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.047+0000 7f9d171ec640 1 -- 172.21.15.29:0/3993309311 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f9cfc0ac560 msgr2=0x7f9cfc0aea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:12.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.048+0000 7f9d171ec640 1 --2- 172.21.15.29:0/3993309311 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f9cfc0ac560 0x7f9cfc0aea70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9d00034000 tx=0x7f9d00008030 comp rx=0 tx=0).stop 2024-04-25T19:44:12.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.048+0000 7f9d171ec640 1 -- 172.21.15.29:0/3993309311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d100aceb0 msgr2=0x7f9d1014a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:12.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.048+0000 7f9d171ec640 1 --2- 172.21.15.29:0/3993309311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d100aceb0 0x7f9d1014a320 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9d0400c820 tx=0x7f9d0400cce0 comp rx=0 tx=0).stop 2024-04-25T19:44:12.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.048+0000 7f9d171ec640 1 -- 172.21.15.29:0/3993309311 shutdown_connections 2024-04-25T19:44:12.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.048+0000 7f9d171ec640 1 --2- 172.21.15.29:0/3993309311 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f9cfc0ac560 0x7f9cfc0aea70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:12.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.048+0000 7f9d171ec640 1 --2- 172.21.15.29:0/3993309311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d10143e30 0x7f9d101e0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:12.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.048+0000 7f9d171ec640 1 --2- 172.21.15.29:0/3993309311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d100aceb0 0x7f9d1014a320 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:12.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.048+0000 7f9d171ec640 1 --2- 172.21.15.29:0/3993309311 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9d100acaa0 0x7f9d1014e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:12.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.048+0000 7f9d171ec640 1 -- 172.21.15.29:0/3993309311 >> 172.21.15.29:0/3993309311 conn(0x7f9d100aafc0 msgr2=0x7f9d10142eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:12.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.048+0000 7f9d171ec640 1 -- 172.21.15.29:0/3993309311 shutdown_connections 2024-04-25T19:44:12.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.048+0000 7f9d171ec640 1 -- 172.21.15.29:0/3993309311 wait complete. 2024-04-25T19:44:12.062 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-25T19:44:12.063 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T19:44:12.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.183+0000 7fdaff510640 1 Processor -- start 2024-04-25T19:44:12.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.183+0000 7fdaff510640 1 -- start start 2024-04-25T19:44:12.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.184+0000 7fdaff510640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdaf8144240 0x7fdaf8146c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:12.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.184+0000 7fdaff510640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdaf8147150 0x7fdaf8151590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:12.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.184+0000 7fdaff510640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdaf8144650 con 0x7fdaf8143e30 2024-04-25T19:44:12.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.184+0000 7fdaff510640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fdaf8147560 con 0x7fdaf8144240 2024-04-25T19:44:12.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.184+0000 7fdaff510640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fdaf80a5a90 con 0x7fdaf8147150 2024-04-25T19:44:12.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.184+0000 7fdafd2ac640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdaf8143e30 0x7fdaf80acaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37994/0 (socket says 172.21.15.29:37994) 2024-04-25T19:44:12.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.184+0000 7fdafd2ac640 1 -- 172.21.15.29:0/3170355686 learned_addr learned my addr 172.21.15.29:0/3170355686 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:12.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.184+0000 7fdafcaab640 1 --2- 172.21.15.29:0/3170355686 >> v2:172.21.15.29:3300/0 conn(0x7fdaf8144240 0x7fdaf8146c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:12.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.185+0000 7fdafdaad640 1 --2- 172.21.15.29:0/3170355686 >> v2:172.21.15.102:3300/0 conn(0x7fdaf8147150 0x7fdaf8151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:12.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.185+0000 7fdaf7fff640 1 -- 172.21.15.29:0/3170355686 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 279839288 0 0) 0x7fdaf0001110 con 0x7fdaf8143e30 2024-04-25T19:44:12.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.185+0000 7fdaf7fff640 1 -- 172.21.15.29:0/3170355686 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdae0003520 con 0x7fdaf8143e30 2024-04-25T19:44:12.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.185+0000 7fdafcaab640 1 -- 172.21.15.29:0/3170355686 >> v2:172.21.15.102:3300/0 conn(0x7fdaf8147150 msgr2=0x7fdaf8151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:12.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.185+0000 7fdafcaab640 1 --2- 172.21.15.29:0/3170355686 >> v2:172.21.15.102:3300/0 conn(0x7fdaf8147150 0x7fdaf8151590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:12.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.185+0000 7fdafcaab640 1 -- 172.21.15.29:0/3170355686 >> v1:172.21.15.29:6789/0 conn(0x7fdaf8143e30 legacy=0x7fdaf80acaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:12.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.185+0000 7fdafcaab640 1 -- 172.21.15.29:0/3170355686 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaf8147790 con 0x7fdaf8144240 2024-04-25T19:44:12.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.185+0000 7fdafdaad640 1 --2- 172.21.15.29:0/3170355686 >> v2:172.21.15.102:3300/0 conn(0x7fdaf8147150 0x7fdaf8151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:44:12.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.186+0000 7fdafcaab640 1 --2- 172.21.15.29:0/3170355686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaf8144240 0x7fdaf8146c10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdaf80addd0 tx=0x7fdaec02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6792e9ba1e137040 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:12.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.186+0000 7fdaf7fff640 1 -- 172.21.15.29:0/3170355686 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaec0315a0 con 0x7fdaf8144240 2024-04-25T19:44:12.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.186+0000 7fdaf7fff640 1 -- 172.21.15.29:0/3170355686 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaec037070 con 0x7fdaf8144240 2024-04-25T19:44:12.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.186+0000 7fdaf7fff640 1 -- 172.21.15.29:0/3170355686 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaec031940 con 0x7fdaf8144240 2024-04-25T19:44:12.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.186+0000 7fdaff510640 1 -- 172.21.15.29:0/3170355686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaf8144240 msgr2=0x7fdaf8146c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:12.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.186+0000 7fdaff510640 1 --2- 172.21.15.29:0/3170355686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaf8144240 0x7fdaf8146c10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdaf80addd0 tx=0x7fdaec02b840 comp rx=0 tx=0).stop 2024-04-25T19:44:12.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.186+0000 7fdaff510640 1 -- 172.21.15.29:0/3170355686 shutdown_connections 2024-04-25T19:44:12.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.186+0000 7fdaff510640 1 --2- 172.21.15.29:0/3170355686 >> v2:172.21.15.102:3300/0 conn(0x7fdaf8147150 0x7fdaf8151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:12.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.186+0000 7fdaff510640 1 --2- 172.21.15.29:0/3170355686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaf8144240 0x7fdaf8146c10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:12.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.186+0000 7fdaff510640 1 -- 172.21.15.29:0/3170355686 >> 172.21.15.29:0/3170355686 conn(0x7fdaf80aafc0 msgr2=0x7fdaf80ab3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:12.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.186+0000 7fdaff510640 1 -- 172.21.15.29:0/3170355686 shutdown_connections 2024-04-25T19:44:12.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.186+0000 7fdaff510640 1 -- 172.21.15.29:0/3170355686 wait complete. 2024-04-25T19:44:12.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.188+0000 7fdaff510640 1 Processor -- start 2024-04-25T19:44:12.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.188+0000 7fdaff510640 1 -- start start 2024-04-25T19:44:12.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.189+0000 7fdaff510640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fdaf8143e30 0x7fdaf81400e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:12.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.189+0000 7fdaff510640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdaf8144240 0x7fdaf8145a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:12.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.189+0000 7fdaff510640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaf8147150 0x7fdaf81e0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:12.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.189+0000 7fdaff510640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf80ad330 con 0x7fdaf8147150 2024-04-25T19:44:12.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.189+0000 7fdaff510640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf80ad1b0 con 0x7fdaf8144240 2024-04-25T19:44:12.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.189+0000 7fdaff510640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fdaf8151ad0 con 0x7fdaf8143e30 2024-04-25T19:44:12.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.189+0000 7fdafd2ac640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fdaf8143e30 0x7fdaf81400e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:12.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.189+0000 7fdafdaad640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaf8147150 0x7fdaf81e0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:12.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.189+0000 7fdafd2ac640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fdaf8143e30 0x7fdaf81400e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34370/0 (socket says 172.21.15.29:34370) 2024-04-25T19:44:12.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.189+0000 7fdafd2ac640 1 -- 172.21.15.29:0/700229204 learned_addr learned my addr 172.21.15.29:0/700229204 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:12.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.189+0000 7fdafcaab640 1 --2- 172.21.15.29:0/700229204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdaf8144240 0x7fdaf8145a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:12.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.189+0000 7fdafd2ac640 1 -- 172.21.15.29:0/700229204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdaf8144240 msgr2=0x7fdaf8145a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:12.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.189+0000 7fdafd2ac640 1 --2- 172.21.15.29:0/700229204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdaf8144240 0x7fdaf8145a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:12.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.189+0000 7fdafd2ac640 1 -- 172.21.15.29:0/700229204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaf8147150 msgr2=0x7fdaf81e0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:12.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.189+0000 7fdafd2ac640 1 --2- 172.21.15.29:0/700229204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaf8147150 0x7fdaf81e0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:12.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.189+0000 7fdafd2ac640 1 -- 172.21.15.29:0/700229204 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaf813fdd0 con 0x7fdaf8143e30 2024-04-25T19:44:12.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.189+0000 7fdafcaab640 1 --2- 172.21.15.29:0/700229204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdaf8144240 0x7fdaf8145a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:44:12.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.189+0000 7fdafdaad640 1 --2- 172.21.15.29:0/700229204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaf8147150 0x7fdaf81e0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:12.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.190+0000 7fdafd2ac640 1 --2- 172.21.15.29:0/700229204 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fdaf8143e30 0x7fdaf81400e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdaf000c3d0 tx=0x7fdaf000c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:12.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.190+0000 7fdaf5ffb640 1 -- 172.21.15.29:0/700229204 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf0017070 con 0x7fdaf8143e30 2024-04-25T19:44:12.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.190+0000 7fdaf5ffb640 1 -- 172.21.15.29:0/700229204 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaf0005ce0 con 0x7fdaf8143e30 2024-04-25T19:44:12.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.190+0000 7fdaff510640 1 -- 172.21.15.29:0/700229204 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaf813dfb0 con 0x7fdaf8143e30 2024-04-25T19:44:12.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.190+0000 7fdaf5ffb640 1 -- 172.21.15.29:0/700229204 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf0007310 con 0x7fdaf8143e30 2024-04-25T19:44:12.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.190+0000 7fdaff510640 1 -- 172.21.15.29:0/700229204 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaf81416d0 con 0x7fdaf8143e30 2024-04-25T19:44:12.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.191+0000 7fdaff510640 1 -- 172.21.15.29:0/700229204 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaf8142e20 con 0x7fdaf8143e30 2024-04-25T19:44:12.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.192+0000 7fdaf5ffb640 1 -- 172.21.15.29:0/700229204 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7fdaf001a070 con 0x7fdaf8143e30 2024-04-25T19:44:12.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.192+0000 7fdaf5ffb640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:12.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.193+0000 7fdaf5ffb640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:12.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.193+0000 7fdaf5ffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:12.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.193+0000 7fdaf5ffb640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:12.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.193+0000 7fdaf5ffb640 1 --2- 172.21.15.29:0/700229204 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fdae40ac530 0x7fdae40aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:12.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.193+0000 7fdaf5ffb640 1 -- 172.21.15.29:0/700229204 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7fdaf00d8780 con 0x7fdaf8143e30 2024-04-25T19:44:12.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.193+0000 7fdafcaab640 1 --2- 172.21.15.29:0/700229204 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fdae40ac530 0x7fdae40aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:12.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.194+0000 7fdafcaab640 1 --2- 172.21.15.29:0/700229204 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fdae40ac530 0x7fdae40aea40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdaec036000 tx=0x7fdaec008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:12.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.197+0000 7fdaf5ffb640 1 -- 172.21.15.29:0/700229204 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdaf004eb80 con 0x7fdaf8143e30 2024-04-25T19:44:12.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:12.423+0000 7fdaff510640 1 -- 172.21.15.29:0/700229204 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fdaf80507b0 con 0x7fdaf8143e30 2024-04-25T19:44:13.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.053+0000 7fdaf5ffb640 1 -- 172.21.15.29:0/700229204 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7fdaf8142e20 con 0x7fdaf8143e30 2024-04-25T19:44:13.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.056+0000 7fdaff510640 10 mgrc shutdown 2024-04-25T19:44:13.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.056+0000 7fdaff510640 1 -- 172.21.15.29:0/700229204 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fdae40ac530 msgr2=0x7fdae40aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:13.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.056+0000 7fdaff510640 1 --2- 172.21.15.29:0/700229204 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fdae40ac530 0x7fdae40aea40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdaec036000 tx=0x7fdaec008030 comp rx=0 tx=0).stop 2024-04-25T19:44:13.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.056+0000 7fdaff510640 1 -- 172.21.15.29:0/700229204 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fdaf8143e30 msgr2=0x7fdaf81400e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:13.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.056+0000 7fdaff510640 1 --2- 172.21.15.29:0/700229204 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fdaf8143e30 0x7fdaf81400e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdaf000c3d0 tx=0x7fdaf000c890 comp rx=0 tx=0).stop 2024-04-25T19:44:13.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.056+0000 7fdaff510640 1 -- 172.21.15.29:0/700229204 shutdown_connections 2024-04-25T19:44:13.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.056+0000 7fdaff510640 1 --2- 172.21.15.29:0/700229204 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fdae40ac530 0x7fdae40aea40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.056+0000 7fdaff510640 1 --2- 172.21.15.29:0/700229204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaf8147150 0x7fdaf81e0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.056+0000 7fdaff510640 1 --2- 172.21.15.29:0/700229204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdaf8144240 0x7fdaf8145a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.056+0000 7fdaff510640 1 --2- 172.21.15.29:0/700229204 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fdaf8143e30 0x7fdaf81400e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.056+0000 7fdaff510640 1 -- 172.21.15.29:0/700229204 >> 172.21.15.29:0/700229204 conn(0x7fdaf80aafc0 msgr2=0x7fdaf80acaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:13.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.056+0000 7fdaff510640 1 -- 172.21.15.29:0/700229204 shutdown_connections 2024-04-25T19:44:13.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.056+0000 7fdaff510640 1 -- 172.21.15.29:0/700229204 wait complete. 2024-04-25T19:44:13.072 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T19:44:13.072 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:44:13.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.190+0000 7f97ac7bb640 1 Processor -- start 2024-04-25T19:44:13.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.191+0000 7f97ac7bb640 1 -- start start 2024-04-25T19:44:13.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.192+0000 7f97ac7bb640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f97a40a56d0 0x7f97a40acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:13.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.192+0000 7f97ac7bb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f97a40ad280 0x7f97a41513f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:13.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.192+0000 7f97ac7bb640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97a4146c60 con 0x7f97a406da90 2024-04-25T19:44:13.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.192+0000 7f97ac7bb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f97a40ad9d0 con 0x7f97a40ad280 2024-04-25T19:44:13.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.192+0000 7f97ac7bb640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f97a40ad690 con 0x7f97a40a56d0 2024-04-25T19:44:13.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.192+0000 7f97a9d56640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f97a40a56d0 0x7f97a40acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:13.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.192+0000 7f97aad58640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f97a40ad280 0x7f97a41513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:13.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.192+0000 7f97aad58640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f97a40ad280 0x7f97a41513f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36464/0 (socket says 172.21.15.29:36464) 2024-04-25T19:44:13.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.192+0000 7f97a9d56640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f97a40a56d0 0x7f97a40acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34384/0 (socket says 172.21.15.29:34384) 2024-04-25T19:44:13.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.192+0000 7f97aad58640 1 -- 172.21.15.29:0/4275234893 learned_addr learned my addr 172.21.15.29:0/4275234893 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:13.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.192+0000 7f97a9d56640 1 -- 172.21.15.29:0/4275234893 >> v2:172.21.15.29:3300/0 conn(0x7f97a40ad280 msgr2=0x7f97a41513f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:13.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.192+0000 7f97a9d56640 1 --2- 172.21.15.29:0/4275234893 >> v2:172.21.15.29:3300/0 conn(0x7f97a40ad280 0x7f97a41513f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.192+0000 7f97a9d56640 1 -- 172.21.15.29:0/4275234893 >> v1:172.21.15.102:6789/0 conn(0x7f97a406da90 legacy=0x7f97a41472d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:13.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.192+0000 7f97a9d56640 1 -- 172.21.15.29:0/4275234893 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a41479e0 con 0x7f97a40a56d0 2024-04-25T19:44:13.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.192+0000 7f97aad58640 1 --2- 172.21.15.29:0/4275234893 >> v2:172.21.15.29:3300/0 conn(0x7f97a40ad280 0x7f97a41513f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:13.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.193+0000 7f97a9d56640 1 --2- 172.21.15.29:0/4275234893 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f97a40a56d0 0x7f97a40acd40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f97940097a0 tx=0x7f979402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ca6d50e27426dec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:13.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.193+0000 7f97a9555640 1 -- 172.21.15.29:0/4275234893 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9794034070 con 0x7f97a40a56d0 2024-04-25T19:44:13.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.193+0000 7f97a9555640 1 -- 172.21.15.29:0/4275234893 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9794030610 con 0x7f97a40a56d0 2024-04-25T19:44:13.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.193+0000 7f97a9555640 1 -- 172.21.15.29:0/4275234893 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97940309e0 con 0x7f97a40a56d0 2024-04-25T19:44:13.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.194+0000 7f97ac7bb640 1 -- 172.21.15.29:0/4275234893 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f97a40a56d0 msgr2=0x7f97a40acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:13.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.194+0000 7f97ac7bb640 1 --2- 172.21.15.29:0/4275234893 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f97a40a56d0 0x7f97a40acd40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f97940097a0 tx=0x7f979402b960 comp rx=0 tx=0).stop 2024-04-25T19:44:13.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.194+0000 7f97ac7bb640 1 -- 172.21.15.29:0/4275234893 shutdown_connections 2024-04-25T19:44:13.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.194+0000 7f97ac7bb640 1 --2- 172.21.15.29:0/4275234893 >> v2:172.21.15.29:3300/0 conn(0x7f97a40ad280 0x7f97a41513f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.194+0000 7f97ac7bb640 1 --2- 172.21.15.29:0/4275234893 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f97a40a56d0 0x7f97a40acd40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.194+0000 7f97ac7bb640 1 -- 172.21.15.29:0/4275234893 >> 172.21.15.29:0/4275234893 conn(0x7f97a40bc3b0 msgr2=0x7f97a40bc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:13.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.194+0000 7f97ac7bb640 1 -- 172.21.15.29:0/4275234893 shutdown_connections 2024-04-25T19:44:13.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.194+0000 7f97ac7bb640 1 -- 172.21.15.29:0/4275234893 wait complete. 2024-04-25T19:44:13.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.195+0000 7f97ac7bb640 1 Processor -- start 2024-04-25T19:44:13.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.196+0000 7f97ac7bb640 1 -- start start 2024-04-25T19:44:13.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.196+0000 7f97ac7bb640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f97a406da90 0x7f97a4148990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:13.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.196+0000 7f97ac7bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a40a56d0 0x7f97a413bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:13.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.196+0000 7f97ac7bb640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97a40ad280 0x7f97a413c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:13.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.197+0000 7f97ac7bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f97a4151ab0 con 0x7f97a40a56d0 2024-04-25T19:44:13.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.197+0000 7f97ac7bb640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f97a4151930 con 0x7f97a40ad280 2024-04-25T19:44:13.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.197+0000 7f97ac7bb640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f97a4151c30 con 0x7f97a406da90 2024-04-25T19:44:13.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.197+0000 7f97aa557640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f97a406da90 0x7f97a4148990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:13.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.197+0000 7f97a9d56640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a40a56d0 0x7f97a413bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:13.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.197+0000 7f97aa557640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f97a406da90 0x7f97a4148990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34396/0 (socket says 172.21.15.29:34396) 2024-04-25T19:44:13.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.197+0000 7f97aa557640 1 -- 172.21.15.29:0/1064026938 learned_addr learned my addr 172.21.15.29:0/1064026938 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.197+0000 7f97aad58640 1 --2- 172.21.15.29:0/1064026938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97a40ad280 0x7f97a413c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.197+0000 7f97aa557640 1 -- 172.21.15.29:0/1064026938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97a40ad280 msgr2=0x7f97a413c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.197+0000 7f97aa557640 1 --2- 172.21.15.29:0/1064026938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97a40ad280 0x7f97a413c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.197+0000 7f97aa557640 1 -- 172.21.15.29:0/1064026938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a40a56d0 msgr2=0x7f97a413bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.197+0000 7f97aa557640 1 --2- 172.21.15.29:0/1064026938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a40a56d0 0x7f97a413bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.197+0000 7f97aa557640 1 -- 172.21.15.29:0/1064026938 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a4144790 con 0x7f97a406da90 2024-04-25T19:44:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.197+0000 7f97aad58640 1 --2- 172.21.15.29:0/1064026938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97a40ad280 0x7f97a413c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:44:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.197+0000 7f97aa557640 1 --2- 172.21.15.29:0/1064026938 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f97a406da90 0x7f97a4148990 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f97a000c3c0 tx=0x7f97a000c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.197+0000 7f979affd640 1 -- 172.21.15.29:0/1064026938 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a00052d0 con 0x7f97a406da90 2024-04-25T19:44:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.197+0000 7f979affd640 1 -- 172.21.15.29:0/1064026938 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97a0005490 con 0x7f97a406da90 2024-04-25T19:44:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.197+0000 7f97ac7bb640 1 -- 172.21.15.29:0/1064026938 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a4149370 con 0x7f97a406da90 2024-04-25T19:44:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.197+0000 7f979affd640 1 -- 172.21.15.29:0/1064026938 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a0007260 con 0x7f97a406da90 2024-04-25T19:44:13.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.197+0000 7f97ac7bb640 1 -- 172.21.15.29:0/1064026938 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a41490b0 con 0x7f97a406da90 2024-04-25T19:44:13.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.198+0000 7f97ac7bb640 1 -- 172.21.15.29:0/1064026938 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a40507b0 con 0x7f97a406da90 2024-04-25T19:44:13.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.200+0000 7f979affd640 1 -- 172.21.15.29:0/1064026938 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f97a0007400 con 0x7f97a406da90 2024-04-25T19:44:13.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.200+0000 7f979affd640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:13.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.201+0000 7f979affd640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:13.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.201+0000 7f979affd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:13.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.201+0000 7f979affd640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:13.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.201+0000 7f979affd640 1 --2- 172.21.15.29:0/1064026938 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f97880ac530 0x7f97880aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:13.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.201+0000 7f979affd640 1 -- 172.21.15.29:0/1064026938 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f97a00d8cd0 con 0x7f97a406da90 2024-04-25T19:44:13.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.201+0000 7f97a9d56640 1 --2- 172.21.15.29:0/1064026938 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f97880ac530 0x7f97880aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:13.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.202+0000 7f97a9d56640 1 --2- 172.21.15.29:0/1064026938 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f97880ac530 0x7f97880aea40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9794036000 tx=0x7f9794008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:13.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.205+0000 7f979affd640 1 -- 172.21.15.29:0/1064026938 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f97a00a3920 con 0x7f97a406da90 2024-04-25T19:44:13.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.427+0000 7f97ac7bb640 1 -- 172.21.15.29:0/1064026938 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97a40b9be0 con 0x7f97a406da90 2024-04-25T19:44:13.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.427+0000 7f979affd640 1 -- 172.21.15.29:0/1064026938 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+4351 (secure 0 0 0) 0x7f97a00a3220 con 0x7f97a406da90 2024-04-25T19:44:13.429 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:13.429 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6809/3582557422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3582557422},{"type":"v1","addr":"172.21.15.29:6809","nonce":3582557422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4234,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/1228868047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1228868047},{"type":"v1","addr":"172.21.15.102:6827","nonce":1228868047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:44:09.410562+0000","modified":"2024-04-25T19:44:12.425418+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6811/844243878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":844243878},{"type":"v1","addr":"172.21.15.29:6811","nonce":844243878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":1}]} 2024-04-25T19:44:13.430 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2024-04-25T19:44:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.429+0000 7f97ac7bb640 10 mgrc shutdown 2024-04-25T19:44:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.429+0000 7f97ac7bb640 1 -- 172.21.15.29:0/1064026938 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f97880ac530 msgr2=0x7f97880aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.429+0000 7f97ac7bb640 1 --2- 172.21.15.29:0/1064026938 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f97880ac530 0x7f97880aea40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9794036000 tx=0x7f9794008030 comp rx=0 tx=0).stop 2024-04-25T19:44:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.429+0000 7f97ac7bb640 1 -- 172.21.15.29:0/1064026938 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f97a406da90 msgr2=0x7f97a4148990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.429+0000 7f97ac7bb640 1 --2- 172.21.15.29:0/1064026938 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f97a406da90 0x7f97a4148990 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f97a000c3c0 tx=0x7f97a000c880 comp rx=0 tx=0).stop 2024-04-25T19:44:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.429+0000 7f97ac7bb640 1 -- 172.21.15.29:0/1064026938 shutdown_connections 2024-04-25T19:44:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.429+0000 7f97ac7bb640 1 --2- 172.21.15.29:0/1064026938 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f97880ac530 0x7f97880aea40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.429+0000 7f97ac7bb640 1 --2- 172.21.15.29:0/1064026938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97a40ad280 0x7f97a413c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.429+0000 7f97ac7bb640 1 --2- 172.21.15.29:0/1064026938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a40a56d0 0x7f97a413bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.429+0000 7f97ac7bb640 1 --2- 172.21.15.29:0/1064026938 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f97a406da90 0x7f97a4148990 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.429+0000 7f97ac7bb640 1 -- 172.21.15.29:0/1064026938 >> 172.21.15.29:0/1064026938 conn(0x7f97a40bc3b0 msgr2=0x7f97a40ab0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.429+0000 7f97ac7bb640 1 -- 172.21.15.29:0/1064026938 shutdown_connections 2024-04-25T19:44:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.429+0000 7f97ac7bb640 1 -- 172.21.15.29:0/1064026938 wait complete. 2024-04-25T19:44:13.441 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:44:13.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.551+0000 7f18b604b640 1 Processor -- start 2024-04-25T19:44:13.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.551+0000 7f18b604b640 1 -- start start 2024-04-25T19:44:13.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.552+0000 7f18b604b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f18b0147030 0x7f18b0147440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:13.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.552+0000 7f18b604b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f18b00a7db0 0x7f18b00a81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:13.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.552+0000 7f18b604b640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18b014a380 con 0x7f18b0073940 2024-04-25T19:44:13.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.552+0000 7f18b604b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f18b014a150 con 0x7f18b00a7db0 2024-04-25T19:44:13.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.552+0000 7f18b604b640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f18b0146ab0 con 0x7f18b0147030 2024-04-25T19:44:13.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.552+0000 7f18affff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f18b00a7db0 0x7f18b00a81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:13.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.552+0000 7f18aeffd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f18b0147030 0x7f18b0147440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:13.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.553+0000 7f18affff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f18b00a7db0 0x7f18b00a81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36476/0 (socket says 172.21.15.29:36476) 2024-04-25T19:44:13.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.553+0000 7f18aeffd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f18b0147030 0x7f18b0147440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34400/0 (socket says 172.21.15.29:34400) 2024-04-25T19:44:13.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.553+0000 7f18affff640 1 -- 172.21.15.29:0/3426288920 learned_addr learned my addr 172.21.15.29:0/3426288920 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:13.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.553+0000 7f18aeffd640 1 -- 172.21.15.29:0/3426288920 >> v2:172.21.15.29:3300/0 conn(0x7f18b00a7db0 msgr2=0x7f18b00a81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:13.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.553+0000 7f18aeffd640 1 --2- 172.21.15.29:0/3426288920 >> v2:172.21.15.29:3300/0 conn(0x7f18b00a7db0 0x7f18b00a81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.553+0000 7f18aeffd640 1 -- 172.21.15.29:0/3426288920 >> v1:172.21.15.102:6789/0 conn(0x7f18b0073940 legacy=0x7f18b006daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:13.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.553+0000 7f18aeffd640 1 -- 172.21.15.29:0/3426288920 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b0147980 con 0x7f18b0147030 2024-04-25T19:44:13.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.553+0000 7f18affff640 1 --2- 172.21.15.29:0/3426288920 >> v2:172.21.15.29:3300/0 conn(0x7f18b00a7db0 0x7f18b00a81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:13.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.554+0000 7f18aeffd640 1 --2- 172.21.15.29:0/3426288920 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f18b0147030 0x7f18b0147440 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f18a40097a0 tx=0x7f18a402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49afeda6d37837cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:13.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.554+0000 7f18ae7fc640 1 -- 172.21.15.29:0/3426288920 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a4035020 con 0x7f18b0147030 2024-04-25T19:44:13.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.554+0000 7f18ae7fc640 1 -- 172.21.15.29:0/3426288920 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18a4031830 con 0x7f18b0147030 2024-04-25T19:44:13.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.554+0000 7f18ae7fc640 1 -- 172.21.15.29:0/3426288920 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a4031c00 con 0x7f18b0147030 2024-04-25T19:44:13.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.554+0000 7f18b604b640 1 -- 172.21.15.29:0/3426288920 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f18b0147030 msgr2=0x7f18b0147440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:13.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.554+0000 7f18b604b640 1 --2- 172.21.15.29:0/3426288920 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f18b0147030 0x7f18b0147440 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f18a40097a0 tx=0x7f18a402b8d0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.555+0000 7f18b604b640 1 -- 172.21.15.29:0/3426288920 shutdown_connections 2024-04-25T19:44:13.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.555+0000 7f18b604b640 1 --2- 172.21.15.29:0/3426288920 >> v2:172.21.15.29:3300/0 conn(0x7f18b00a7db0 0x7f18b00a81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.555+0000 7f18b604b640 1 --2- 172.21.15.29:0/3426288920 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f18b0147030 0x7f18b0147440 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.555+0000 7f18b604b640 1 -- 172.21.15.29:0/3426288920 >> 172.21.15.29:0/3426288920 conn(0x7f18b00a9390 msgr2=0x7f18b00a97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:13.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.555+0000 7f18b604b640 1 -- 172.21.15.29:0/3426288920 shutdown_connections 2024-04-25T19:44:13.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.555+0000 7f18b604b640 1 -- 172.21.15.29:0/3426288920 wait complete. 2024-04-25T19:44:13.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.556+0000 7f18b604b640 1 Processor -- start 2024-04-25T19:44:13.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.556+0000 7f18b604b640 1 -- start start 2024-04-25T19:44:13.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.557+0000 7f18b604b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18b0073940 0x7f18b01d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:13.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.557+0000 7f18b604b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f18b00a7db0 0x7f18b00a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:13.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.557+0000 7f18b604b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18b0147030 0x7f18b01dd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:13.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.557+0000 7f18b604b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f18b006e200 con 0x7f18b0147030 2024-04-25T19:44:13.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.557+0000 7f18b604b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f18b0073f60 con 0x7f18b0073940 2024-04-25T19:44:13.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.557+0000 7f18b604b640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f18b006e380 con 0x7f18b00a7db0 2024-04-25T19:44:13.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.557+0000 7f18aeffd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f18b00a7db0 0x7f18b00a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:13.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.557+0000 7f18aeffd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f18b00a7db0 0x7f18b00a5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34414/0 (socket says 172.21.15.29:34414) 2024-04-25T19:44:13.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.557+0000 7f18aeffd640 1 -- 172.21.15.29:0/2138448318 learned_addr learned my addr 172.21.15.29:0/2138448318 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:13.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.557+0000 7f18affff640 1 --2- 172.21.15.29:0/2138448318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18b0147030 0x7f18b01dd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:13.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.557+0000 7f18af7fe640 1 --2- 172.21.15.29:0/2138448318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18b0073940 0x7f18b01d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:13.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.558+0000 7f18aeffd640 1 -- 172.21.15.29:0/2138448318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18b0073940 msgr2=0x7f18b01d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:13.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.558+0000 7f18aeffd640 1 --2- 172.21.15.29:0/2138448318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18b0073940 0x7f18b01d08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.558+0000 7f18aeffd640 1 -- 172.21.15.29:0/2138448318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18b0147030 msgr2=0x7f18b01dd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:13.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.558+0000 7f18aeffd640 1 --2- 172.21.15.29:0/2138448318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18b0147030 0x7f18b01dd7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.558+0000 7f18aeffd640 1 -- 172.21.15.29:0/2138448318 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b01435d0 con 0x7f18b00a7db0 2024-04-25T19:44:13.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.558+0000 7f18affff640 1 --2- 172.21.15.29:0/2138448318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18b0147030 0x7f18b01dd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:13.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.558+0000 7f18aeffd640 1 --2- 172.21.15.29:0/2138448318 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f18b00a7db0 0x7f18b00a5a60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f18a40098c0 tx=0x7f18a40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:13.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.558+0000 7f1897fff640 1 -- 172.21.15.29:0/2138448318 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a4040070 con 0x7f18b00a7db0 2024-04-25T19:44:13.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.558+0000 7f1897fff640 1 -- 172.21.15.29:0/2138448318 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18a403e070 con 0x7f18b00a7db0 2024-04-25T19:44:13.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.558+0000 7f18b604b640 1 -- 172.21.15.29:0/2138448318 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18b01d0de0 con 0x7f18b00a7db0 2024-04-25T19:44:13.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.558+0000 7f1897fff640 1 -- 172.21.15.29:0/2138448318 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a4035040 con 0x7f18b00a7db0 2024-04-25T19:44:13.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.558+0000 7f18b604b640 1 -- 172.21.15.29:0/2138448318 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f18b01d2bd0 con 0x7f18b00a7db0 2024-04-25T19:44:13.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.559+0000 7f18b604b640 1 -- 172.21.15.29:0/2138448318 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18b00507b0 con 0x7f18b00a7db0 2024-04-25T19:44:13.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.561+0000 7f1897fff640 1 -- 172.21.15.29:0/2138448318 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f18a4038020 con 0x7f18b00a7db0 2024-04-25T19:44:13.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.561+0000 7f1897fff640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:13.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.562+0000 7f1897fff640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:13.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.562+0000 7f1897fff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:13.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.562+0000 7f1897fff640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:13.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.562+0000 7f1897fff640 1 --2- 172.21.15.29:0/2138448318 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f189c0ac560 0x7f189c0aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:13.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.562+0000 7f1897fff640 1 -- 172.21.15.29:0/2138448318 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f18a40faa70 con 0x7f18b00a7db0 2024-04-25T19:44:13.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.562+0000 7f18af7fe640 1 --2- 172.21.15.29:0/2138448318 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f189c0ac560 0x7f189c0aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:13.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.562+0000 7f18af7fe640 1 --2- 172.21.15.29:0/2138448318 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f189c0ac560 0x7f189c0aea70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f18a0000f60 tx=0x7f18a0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:13.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.565+0000 7f1897fff640 1 -- 172.21.15.29:0/2138448318 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f18a407fb80 con 0x7f18b00a7db0 2024-04-25T19:44:13.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.751+0000 7f18b604b640 1 -- 172.21.15.29:0/2138448318 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f18b0147760 con 0x7f18b00a7db0 2024-04-25T19:44:13.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.752+0000 7f1897fff640 1 -- 172.21.15.29:0/2138448318 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+13512 (secure 0 0 0) 0x7f18a4031800 con 0x7f18b00a7db0 2024-04-25T19:44:13.753 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:13.753 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"1b3c39b5-e787-409e-b0bd-f4f491c89d72","created":"2024-04-25T19:43:47.854697+0000","modified":"2024-04-25T19:44:11.035019+0000","last_up_change":"2024-04-25T19:43:57.853237+0000","last_in_change":"2024-04-25T19:43:49.022776+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:43:58.096700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T19:44:03.617011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:44:07.890493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2024-04-25T19:44:08.355570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d805223-ee84-494f-bd31-59091906aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6801","nonce":695704147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6803","nonce":695704147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6807","nonce":695704147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6805","nonce":695704147}]},"public_addr":"172.21.15.29:6801/695704147","cluster_addr":"172.21.15.29:6803/695704147","heartbeat_back_addr":"172.21.15.29:6807/695704147","heartbeat_front_addr":"172.21.15.29:6805/695704147","state":["exists","up"]},{"osd":1,"uuid":"607c5038-38af-440f-8371-e71758549920","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6814","nonce":3135821452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6817","nonce":3135821452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6823","nonce":3135821452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6821","nonce":3135821452}]},"public_addr":"172.21.15.102:6814/3135821452","cluster_addr":"172.21.15.102:6817/3135821452","heartbeat_back_addr":"172.21.15.102:6823/3135821452","heartbeat_front_addr":"172.21.15.102:6821/3135821452","state":["exists","up"]},{"osd":2,"uuid":"6859ae76-54bb-4c55-b0dc-5e2735c0d31c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6801","nonce":2071844144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6803","nonce":2071844144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6811","nonce":2071844144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6806","nonce":2071844144}]},"public_addr":"172.21.15.102:6801/2071844144","cluster_addr":"172.21.15.102:6803/2071844144","heartbeat_back_addr":"172.21.15.102:6811/2071844144","heartbeat_front_addr":"172.21.15.102:6806/2071844144","state":["exists","up"]},{"osd":3,"uuid":"a98b7b88-5206-4f22-84a6-78212e0d5f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6805","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6807","nonce":339944984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6810","nonce":339944984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6820","nonce":339944984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6815","nonce":339944984}]},"public_addr":"172.21.15.102:6807/339944984","cluster_addr":"172.21.15.102:6810/339944984","heartbeat_back_addr":"172.21.15.102:6820/339944984","heartbeat_front_addr":"172.21.15.102:6815/339944984","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.121529+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.201155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.264387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.198713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:44:13.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.754+0000 7f18b604b640 10 mgrc shutdown 2024-04-25T19:44:13.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.754+0000 7f18b604b640 1 -- 172.21.15.29:0/2138448318 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f189c0ac560 msgr2=0x7f189c0aea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:13.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.754+0000 7f18b604b640 1 --2- 172.21.15.29:0/2138448318 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f189c0ac560 0x7f189c0aea70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f18a0000f60 tx=0x7f18a0009040 comp rx=0 tx=0).stop 2024-04-25T19:44:13.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.754+0000 7f18b604b640 1 -- 172.21.15.29:0/2138448318 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f18b00a7db0 msgr2=0x7f18b00a5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:13.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.754+0000 7f18b604b640 1 --2- 172.21.15.29:0/2138448318 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f18b00a7db0 0x7f18b00a5a60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f18a40098c0 tx=0x7f18a40046c0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.754+0000 7f18b604b640 1 -- 172.21.15.29:0/2138448318 shutdown_connections 2024-04-25T19:44:13.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.754+0000 7f18b604b640 1 --2- 172.21.15.29:0/2138448318 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f189c0ac560 0x7f189c0aea70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.754+0000 7f18b604b640 1 --2- 172.21.15.29:0/2138448318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18b0147030 0x7f18b01dd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.754+0000 7f18b604b640 1 --2- 172.21.15.29:0/2138448318 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f18b00a7db0 0x7f18b00a5a60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.754+0000 7f18b604b640 1 --2- 172.21.15.29:0/2138448318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18b0073940 0x7f18b01d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.754+0000 7f18b604b640 1 -- 172.21.15.29:0/2138448318 >> 172.21.15.29:0/2138448318 conn(0x7f18b00a9390 msgr2=0x7f18b00684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:13.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.754+0000 7f18b604b640 1 -- 172.21.15.29:0/2138448318 shutdown_connections 2024-04-25T19:44:13.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.754+0000 7f18b604b640 1 -- 172.21.15.29:0/2138448318 wait complete. 2024-04-25T19:44:13.766 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T19:44:13.766 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T19:44:13.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.876+0000 7f3b9f3bd640 1 Processor -- start 2024-04-25T19:44:13.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.876+0000 7f3b9f3bd640 1 -- start start 2024-04-25T19:44:13.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.877+0000 7f3b9f3bd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3b98143e30 0x7f3b98144240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.877+0000 7f3b9f3bd640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3b980acaa0 0x7f3b980aceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.877+0000 7f3b9f3bd640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b980a5a90 con 0x7f3b98146c10 2024-04-25T19:44:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.877+0000 7f3b9f3bd640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f3b98147790 con 0x7f3b980acaa0 2024-04-25T19:44:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.877+0000 7f3b9f3bd640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f3b98147a50 con 0x7f3b98143e30 2024-04-25T19:44:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.877+0000 7f3b9d159640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3b98143e30 0x7f3b98144240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.877+0000 7f3b9d159640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3b98143e30 0x7f3b98144240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34418/0 (socket says 172.21.15.29:34418) 2024-04-25T19:44:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.877+0000 7f3b9d159640 1 -- 172.21.15.29:0/3515004218 learned_addr learned my addr 172.21.15.29:0/3515004218 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.877+0000 7f3b9c958640 1 --2- 172.21.15.29:0/3515004218 >> v2:172.21.15.102:3300/0 conn(0x7f3b980acaa0 0x7f3b980aceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.877+0000 7f3b9d159640 1 -- 172.21.15.29:0/3515004218 >> v2:172.21.15.102:3300/0 conn(0x7f3b980acaa0 msgr2=0x7f3b980aceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.877+0000 7f3b9d159640 1 --2- 172.21.15.29:0/3515004218 >> v2:172.21.15.102:3300/0 conn(0x7f3b980acaa0 0x7f3b980aceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.877+0000 7f3b9d159640 1 -- 172.21.15.29:0/3515004218 >> v1:172.21.15.102:6789/0 conn(0x7f3b98146c10 legacy=0x7f3b9814a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.877+0000 7f3b9d159640 1 -- 172.21.15.29:0/3515004218 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b98147020 con 0x7f3b98143e30 2024-04-25T19:44:13.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.878+0000 7f3b9d159640 1 --2- 172.21.15.29:0/3515004218 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3b98143e30 0x7f3b98144240 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3b940098a0 tx=0x7f3b9402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b05ad80976f42ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:13.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.878+0000 7f3b8ffff640 1 -- 172.21.15.29:0/3515004218 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b94034070 con 0x7f3b98143e30 2024-04-25T19:44:13.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.878+0000 7f3b8ffff640 1 -- 172.21.15.29:0/3515004218 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b94030840 con 0x7f3b98143e30 2024-04-25T19:44:13.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.878+0000 7f3b8ffff640 1 -- 172.21.15.29:0/3515004218 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b94030c10 con 0x7f3b98143e30 2024-04-25T19:44:13.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.879+0000 7f3b9f3bd640 1 -- 172.21.15.29:0/3515004218 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3b98143e30 msgr2=0x7f3b98144240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:13.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.879+0000 7f3b9f3bd640 1 --2- 172.21.15.29:0/3515004218 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3b98143e30 0x7f3b98144240 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3b940098a0 tx=0x7f3b9402b980 comp rx=0 tx=0).stop 2024-04-25T19:44:13.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.879+0000 7f3b9f3bd640 1 -- 172.21.15.29:0/3515004218 shutdown_connections 2024-04-25T19:44:13.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.879+0000 7f3b9f3bd640 1 --2- 172.21.15.29:0/3515004218 >> v2:172.21.15.102:3300/0 conn(0x7f3b980acaa0 0x7f3b980aceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.879+0000 7f3b9f3bd640 1 --2- 172.21.15.29:0/3515004218 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3b98143e30 0x7f3b98144240 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.879+0000 7f3b9f3bd640 1 -- 172.21.15.29:0/3515004218 >> 172.21.15.29:0/3515004218 conn(0x7f3b980aafc0 msgr2=0x7f3b980ab3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:13.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.879+0000 7f3b9f3bd640 1 -- 172.21.15.29:0/3515004218 shutdown_connections 2024-04-25T19:44:13.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.879+0000 7f3b9f3bd640 1 -- 172.21.15.29:0/3515004218 wait complete. 2024-04-25T19:44:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.880+0000 7f3b9f3bd640 1 Processor -- start 2024-04-25T19:44:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.880+0000 7f3b9f3bd640 1 -- start start 2024-04-25T19:44:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.881+0000 7f3b9f3bd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b980acaa0 0x7f3b980a7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.881+0000 7f3b9f3bd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b98143e30 0x7f3b981e06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.881+0000 7f3b9f3bd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3b98146c10 0x7f3b9813fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.881+0000 7f3b9f3bd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9814aa90 con 0x7f3b980acaa0 2024-04-25T19:44:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.881+0000 7f3b9f3bd640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9814a910 con 0x7f3b98143e30 2024-04-25T19:44:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.881+0000 7f3b9f3bd640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f3b9814ac10 con 0x7f3b98146c10 2024-04-25T19:44:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.882+0000 7f3b9d159640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b980acaa0 0x7f3b980a7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.882+0000 7f3b9d95a640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3b98146c10 0x7f3b9813fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.882+0000 7f3b9d159640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b980acaa0 0x7f3b980a7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36490/0 (socket says 172.21.15.29:36490) 2024-04-25T19:44:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.882+0000 7f3b9d159640 1 -- 172.21.15.29:0/172477953 learned_addr learned my addr 172.21.15.29:0/172477953 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.882+0000 7f3b9c958640 1 --2- 172.21.15.29:0/172477953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b98143e30 0x7f3b981e06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.882+0000 7f3b9d159640 1 -- 172.21.15.29:0/172477953 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3b98146c10 msgr2=0x7f3b9813fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.882+0000 7f3b9d159640 1 --2- 172.21.15.29:0/172477953 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3b98146c10 0x7f3b9813fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.882+0000 7f3b9d159640 1 -- 172.21.15.29:0/172477953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b98143e30 msgr2=0x7f3b981e06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.882+0000 7f3b9d159640 1 --2- 172.21.15.29:0/172477953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b98143e30 0x7f3b981e06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.882+0000 7f3b9d159640 1 -- 172.21.15.29:0/172477953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b980af690 con 0x7f3b980acaa0 2024-04-25T19:44:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.882+0000 7f3b9d95a640 1 --2- 172.21.15.29:0/172477953 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3b98146c10 0x7f3b9813fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.882+0000 7f3b9d159640 1 --2- 172.21.15.29:0/172477953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b980acaa0 0x7f3b980a7790 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3b94000c00 tx=0x7f3b940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.882+0000 7f3b8dffb640 1 -- 172.21.15.29:0/172477953 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b94034040 con 0x7f3b980acaa0 2024-04-25T19:44:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.882+0000 7f3b8dffb640 1 -- 172.21.15.29:0/172477953 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b94039d90 con 0x7f3b980acaa0 2024-04-25T19:44:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.882+0000 7f3b9f3bd640 1 -- 172.21.15.29:0/172477953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b981d3a90 con 0x7f3b980acaa0 2024-04-25T19:44:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.882+0000 7f3b8dffb640 1 -- 172.21.15.29:0/172477953 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b94043630 con 0x7f3b980acaa0 2024-04-25T19:44:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.883+0000 7f3b9f3bd640 1 -- 172.21.15.29:0/172477953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b980b32b0 con 0x7f3b980acaa0 2024-04-25T19:44:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.884+0000 7f3b9f3bd640 1 -- 172.21.15.29:0/172477953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b980b2520 con 0x7f3b980acaa0 2024-04-25T19:44:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.885+0000 7f3b8dffb640 1 -- 172.21.15.29:0/172477953 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f3b9404a060 con 0x7f3b980acaa0 2024-04-25T19:44:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.885+0000 7f3b8dffb640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:13.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.885+0000 7f3b8dffb640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:13.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.885+0000 7f3b8dffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:13.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.885+0000 7f3b8dffb640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:13.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.885+0000 7f3b8dffb640 1 --2- 172.21.15.29:0/172477953 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f3b7c0ac530 0x7f3b7c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:13.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.886+0000 7f3b8dffb640 1 -- 172.21.15.29:0/172477953 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f3b940fb640 con 0x7f3b980acaa0 2024-04-25T19:44:13.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.886+0000 7f3b9c958640 1 --2- 172.21.15.29:0/172477953 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f3b7c0ac530 0x7f3b7c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:13.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.886+0000 7f3b9c958640 1 --2- 172.21.15.29:0/172477953 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f3b7c0ac530 0x7f3b7c0aea40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3b981de530 tx=0x7f3b88009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:13.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:13.889+0000 7f3b8dffb640 1 -- 172.21.15.29:0/172477953 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3b9403e070 con 0x7f3b980acaa0 2024-04-25T19:44:14.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.070+0000 7f3b9f3bd640 20 mgrc start_command cmd: [{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}] 2024-04-25T19:44:14.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.070+0000 7f3b9f3bd640 1 -- 172.21.15.29:0/172477953 --> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3b981e0fd0 con 0x7f3b7c0ac530 2024-04-25T19:44:14.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.078+0000 7f3b8dffb640 1 -- 172.21.15.29:0/172477953 <== mgr.4108 v2:172.21.15.102:6824/1439704316 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257200 (secure 0 0 0) 0x7f3b981e0fd0 con 0x7f3b7c0ac530 2024-04-25T19:44:14.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.078+0000 7f3b8dffb640 20 mgrc handle_command_reply tid 0 r 0 2024-04-25T19:44:14.082 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:14.086 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-04-25T19:44:14.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.080+0000 7f3b9f3bd640 10 mgrc shutdown 2024-04-25T19:44:14.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.080+0000 7f3b9f3bd640 1 -- 172.21.15.29:0/172477953 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f3b7c0ac530 msgr2=0x7f3b7c0aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:14.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.080+0000 7f3b9f3bd640 1 --2- 172.21.15.29:0/172477953 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f3b7c0ac530 0x7f3b7c0aea40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3b981de530 tx=0x7f3b88009190 comp rx=0 tx=0).stop 2024-04-25T19:44:14.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.081+0000 7f3b9f3bd640 1 -- 172.21.15.29:0/172477953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b980acaa0 msgr2=0x7f3b980a7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:14.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.081+0000 7f3b9f3bd640 1 --2- 172.21.15.29:0/172477953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b980acaa0 0x7f3b980a7790 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3b94000c00 tx=0x7f3b940046c0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.081+0000 7f3b9f3bd640 1 -- 172.21.15.29:0/172477953 shutdown_connections 2024-04-25T19:44:14.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.081+0000 7f3b9f3bd640 1 --2- 172.21.15.29:0/172477953 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f3b7c0ac530 0x7f3b7c0aea40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.081+0000 7f3b9f3bd640 1 --2- 172.21.15.29:0/172477953 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3b98146c10 0x7f3b9813fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.081+0000 7f3b9f3bd640 1 --2- 172.21.15.29:0/172477953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b98143e30 0x7f3b981e06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.081+0000 7f3b9f3bd640 1 --2- 172.21.15.29:0/172477953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b980acaa0 0x7f3b980a7790 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.081+0000 7f3b9f3bd640 1 -- 172.21.15.29:0/172477953 >> 172.21.15.29:0/172477953 conn(0x7f3b980aafc0 msgr2=0x7f3b980ba8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:14.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.081+0000 7f3b9f3bd640 1 -- 172.21.15.29:0/172477953 shutdown_connections 2024-04-25T19:44:14.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.081+0000 7f3b9f3bd640 1 -- 172.21.15.29:0/172477953 wait complete. 2024-04-25T19:44:14.093 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-04-25T19:44:12.082579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":72,"acting":72,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":113952,"kb_used_data":1704,"kb_used_omap":6,"kb_used_meta":112185,"kb_avail":377373408,"statfs":{"total":386547056640,"available":386430369792,"internally_reserved":0,"allocated":1744896,"data_stored":1353300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":114878230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.064596"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.024732+0000","last_change":"2024-04-25T19:44:08.024732+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.024732+0000","last_undegraded":"2024-04-25T19:44:08.024732+0000","last_fullsized":"2024-04-25T19:44:08.024732+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.023362+0000","last_change":"2024-04-25T19:44:08.023362+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.023362+0000","last_undegraded":"2024-04-25T19:44:08.023362+0000","last_fullsized":"2024-04-25T19:44:08.023362+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.023852+0000","last_change":"2024-04-25T19:44:08.023852+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.023852+0000","last_undegraded":"2024-04-25T19:44:08.023852+0000","last_fullsized":"2024-04-25T19:44:08.023852+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.020046+0000","last_change":"2024-04-25T19:44:08.020046+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.020046+0000","last_undegraded":"2024-04-25T19:44:08.020046+0000","last_fullsized":"2024-04-25T19:44:08.020046+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.023365+0000","last_change":"2024-04-25T19:44:08.023365+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.023365+0000","last_undegraded":"2024-04-25T19:44:08.023365+0000","last_fullsized":"2024-04-25T19:44:08.023365+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.023009+0000","last_change":"2024-04-25T19:44:08.023009+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.023009+0000","last_undegraded":"2024-04-25T19:44:08.023009+0000","last_fullsized":"2024-04-25T19:44:08.023009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.025866+0000","last_change":"2024-04-25T19:44:08.025866+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.025866+0000","last_undegraded":"2024-04-25T19:44:08.025866+0000","last_fullsized":"2024-04-25T19:44:08.025866+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.024873+0000","last_change":"2024-04-25T19:44:08.024873+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.024873+0000","last_undegraded":"2024-04-25T19:44:08.024873+0000","last_fullsized":"2024-04-25T19:44:08.024873+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:08.015661+0000","last_change":"2024-04-25T19:44:08.015661+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.015661+0000","last_undegraded":"2024-04-25T19:44:08.015661+0000","last_fullsized":"2024-04-25T19:44:08.015661+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.025019+0000","last_change":"2024-04-25T19:44:08.025019+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.025019+0000","last_undegraded":"2024-04-25T19:44:08.025019+0000","last_fullsized":"2024-04-25T19:44:08.025019+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.022999+0000","last_change":"2024-04-25T19:44:08.022999+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.022999+0000","last_undegraded":"2024-04-25T19:44:08.022999+0000","last_fullsized":"2024-04-25T19:44:08.022999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.023054+0000","last_change":"2024-04-25T19:44:08.023054+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.023054+0000","last_undegraded":"2024-04-25T19:44:08.023054+0000","last_fullsized":"2024-04-25T19:44:08.023054+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"12'72","reported_seq":110,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-25T19:44:08.021899+0000","last_change":"2024-04-25T19:44:00.045570+0000","last_active":"2024-04-25T19:44:08.021899+0000","last_peered":"2024-04-25T19:44:08.021899+0000","last_clean":"2024-04-25T19:44:08.021899+0000","last_became_active":"2024-04-25T19:44:00.044414+0000","last_became_peered":"2024-04-25T19:44:00.044414+0000","last_unstale":"2024-04-25T19:44:08.021899+0000","last_undegraded":"2024-04-25T19:44:08.021899+0000","last_fullsized":"2024-04-25T19:44:08.021899+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:43:58.854910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:43:58.854910+0000","last_clean_scrub_stamp":"2024-04-25T19:43:58.854910+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:36.813186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-25T19:44:08.026100+0000","last_change":"2024-04-25T19:44:04.907674+0000","last_active":"2024-04-25T19:44:08.026100+0000","last_peered":"2024-04-25T19:44:08.026100+0000","last_clean":"2024-04-25T19:44:08.026100+0000","last_became_active":"2024-04-25T19:44:04.907259+0000","last_became_peered":"2024-04-25T19:44:04.907259+0000","last_unstale":"2024-04-25T19:44:08.026100+0000","last_undegraded":"2024-04-25T19:44:08.026100+0000","last_fullsized":"2024-04-25T19:44:08.026100+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:58.023878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.024452+0000","last_change":"2024-04-25T19:44:08.024452+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.024452+0000","last_undegraded":"2024-04-25T19:44:08.024452+0000","last_fullsized":"2024-04-25T19:44:08.024452+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-25T19:44:08.024081+0000","last_change":"2024-04-25T19:44:04.904786+0000","last_active":"2024-04-25T19:44:08.024081+0000","last_peered":"2024-04-25T19:44:08.024081+0000","last_clean":"2024-04-25T19:44:08.024081+0000","last_became_active":"2024-04-25T19:44:04.904414+0000","last_became_peered":"2024-04-25T19:44:04.904414+0000","last_unstale":"2024-04-25T19:44:08.024081+0000","last_undegraded":"2024-04-25T19:44:08.024081+0000","last_fullsized":"2024-04-25T19:44:08.024081+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:48:36.719208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:08.015661+0000","last_change":"2024-04-25T19:44:08.015661+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.015661+0000","last_undegraded":"2024-04-25T19:44:08.015661+0000","last_fullsized":"2024-04-25T19:44:08.015661+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-25T19:44:08.025908+0000","last_change":"2024-04-25T19:44:04.904405+0000","last_active":"2024-04-25T19:44:08.025908+0000","last_peered":"2024-04-25T19:44:08.025908+0000","last_clean":"2024-04-25T19:44:08.025908+0000","last_became_active":"2024-04-25T19:44:04.904045+0000","last_became_peered":"2024-04-25T19:44:04.904045+0000","last_unstale":"2024-04-25T19:44:08.025908+0000","last_undegraded":"2024-04-25T19:44:08.025908+0000","last_fullsized":"2024-04-25T19:44:08.025908+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:01:59.962699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:08.015661+0000","last_change":"2024-04-25T19:44:08.015661+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.015661+0000","last_undegraded":"2024-04-25T19:44:08.015661+0000","last_fullsized":"2024-04-25T19:44:08.015661+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-25T19:44:08.025289+0000","last_change":"2024-04-25T19:44:04.907093+0000","last_active":"2024-04-25T19:44:08.025289+0000","last_peered":"2024-04-25T19:44:08.025289+0000","last_clean":"2024-04-25T19:44:08.025289+0000","last_became_active":"2024-04-25T19:44:04.906787+0000","last_became_peered":"2024-04-25T19:44:04.906787+0000","last_unstale":"2024-04-25T19:44:08.025289+0000","last_undegraded":"2024-04-25T19:44:08.025289+0000","last_fullsized":"2024-04-25T19:44:08.025289+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:28:59.374809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.022941+0000","last_change":"2024-04-25T19:44:08.022941+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.022941+0000","last_undegraded":"2024-04-25T19:44:08.022941+0000","last_fullsized":"2024-04-25T19:44:08.022941+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.023826+0000","last_change":"2024-04-25T19:44:08.023826+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.023826+0000","last_undegraded":"2024-04-25T19:44:08.023826+0000","last_fullsized":"2024-04-25T19:44:08.023826+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-25T19:44:05.900424+0000","last_change":"2024-04-25T19:44:04.905598+0000","last_active":"2024-04-25T19:44:05.900424+0000","last_peered":"2024-04-25T19:44:05.900424+0000","last_clean":"2024-04-25T19:44:05.900424+0000","last_became_active":"2024-04-25T19:44:04.904778+0000","last_became_peered":"2024-04-25T19:44:04.904778+0000","last_unstale":"2024-04-25T19:44:05.900424+0000","last_undegraded":"2024-04-25T19:44:05.900424+0000","last_fullsized":"2024-04-25T19:44:05.900424+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:03:40.381610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:08.015661+0000","last_change":"2024-04-25T19:44:08.015661+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.015661+0000","last_undegraded":"2024-04-25T19:44:08.015661+0000","last_fullsized":"2024-04-25T19:44:08.015661+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-25T19:44:08.026689+0000","last_change":"2024-04-25T19:44:04.904842+0000","last_active":"2024-04-25T19:44:08.026689+0000","last_peered":"2024-04-25T19:44:08.026689+0000","last_clean":"2024-04-25T19:44:08.026689+0000","last_became_active":"2024-04-25T19:44:04.904449+0000","last_became_peered":"2024-04-25T19:44:04.904449+0000","last_unstale":"2024-04-25T19:44:08.026689+0000","last_undegraded":"2024-04-25T19:44:08.026689+0000","last_fullsized":"2024-04-25T19:44:08.026689+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:38.322685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.023207+0000","last_change":"2024-04-25T19:44:08.023207+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.023207+0000","last_undegraded":"2024-04-25T19:44:08.023207+0000","last_fullsized":"2024-04-25T19:44:08.023207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-25T19:44:05.900456+0000","last_change":"2024-04-25T19:44:04.904847+0000","last_active":"2024-04-25T19:44:05.900456+0000","last_peered":"2024-04-25T19:44:05.900456+0000","last_clean":"2024-04-25T19:44:05.900456+0000","last_became_active":"2024-04-25T19:44:04.904399+0000","last_became_peered":"2024-04-25T19:44:04.904399+0000","last_unstale":"2024-04-25T19:44:05.900456+0000","last_undegraded":"2024-04-25T19:44:05.900456+0000","last_fullsized":"2024-04-25T19:44:05.900456+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:31.155970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.020091+0000","last_change":"2024-04-25T19:44:08.020091+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.020091+0000","last_undegraded":"2024-04-25T19:44:08.020091+0000","last_fullsized":"2024-04-25T19:44:08.020091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-25T19:44:08.023798+0000","last_change":"2024-04-25T19:44:04.907462+0000","last_active":"2024-04-25T19:44:08.023798+0000","last_peered":"2024-04-25T19:44:08.023798+0000","last_clean":"2024-04-25T19:44:08.023798+0000","last_became_active":"2024-04-25T19:44:04.907134+0000","last_became_peered":"2024-04-25T19:44:04.907134+0000","last_unstale":"2024-04-25T19:44:08.023798+0000","last_undegraded":"2024-04-25T19:44:08.023798+0000","last_fullsized":"2024-04-25T19:44:08.023798+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:05:53.288205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.025422+0000","last_change":"2024-04-25T19:44:08.025422+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.025422+0000","last_undegraded":"2024-04-25T19:44:08.025422+0000","last_fullsized":"2024-04-25T19:44:08.025422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.024701+0000","last_change":"2024-04-25T19:44:08.024701+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.024701+0000","last_undegraded":"2024-04-25T19:44:08.024701+0000","last_fullsized":"2024-04-25T19:44:08.024701+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.024286+0000","last_change":"2024-04-25T19:44:08.024286+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.024286+0000","last_undegraded":"2024-04-25T19:44:08.024286+0000","last_fullsized":"2024-04-25T19:44:08.024286+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.020109+0000","last_change":"2024-04-25T19:44:08.020109+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.020109+0000","last_undegraded":"2024-04-25T19:44:08.020109+0000","last_fullsized":"2024-04-25T19:44:08.020109+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.023750+0000","last_change":"2024-04-25T19:44:08.023750+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.023750+0000","last_undegraded":"2024-04-25T19:44:08.023750+0000","last_fullsized":"2024-04-25T19:44:08.023750+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.020083+0000","last_change":"2024-04-25T19:44:08.020083+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.020083+0000","last_undegraded":"2024-04-25T19:44:08.020083+0000","last_fullsized":"2024-04-25T19:44:08.020083+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.020080+0000","last_change":"2024-04-25T19:44:08.020080+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.020080+0000","last_undegraded":"2024-04-25T19:44:08.020080+0000","last_fullsized":"2024-04-25T19:44:08.020080+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.025637+0000","last_change":"2024-04-25T19:44:08.025637+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.025637+0000","last_undegraded":"2024-04-25T19:44:08.025637+0000","last_fullsized":"2024-04-25T19:44:08.025637+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.020045+0000","last_change":"2024-04-25T19:44:08.020045+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.020045+0000","last_undegraded":"2024-04-25T19:44:08.020045+0000","last_fullsized":"2024-04-25T19:44:08.020045+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.026418+0000","last_change":"2024-04-25T19:44:08.026418+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.026418+0000","last_undegraded":"2024-04-25T19:44:08.026418+0000","last_fullsized":"2024-04-25T19:44:08.026418+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:08.015661+0000","last_change":"2024-04-25T19:44:08.015661+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.015661+0000","last_undegraded":"2024-04-25T19:44:08.015661+0000","last_fullsized":"2024-04-25T19:44:08.015661+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T19:44:09.017642+0000","last_change":"2024-04-25T19:44:09.017642+0000","last_active":"2024-04-25T19:44:09.017642+0000","last_peered":"2024-04-25T19:44:09.017642+0000","last_clean":"2024-04-25T19:44:09.017642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:09.017642+0000","last_undegraded":"2024-04-25T19:44:09.017642+0000","last_fullsized":"2024-04-25T19:44:09.017642+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-25T19:44:08.021019+0000","last_change":"2024-04-25T19:44:08.021019+0000","last_active":"2024-04-25T19:44:08.015661+0000","last_peered":"2024-04-25T19:44:08.015661+0000","last_clean":"2024-04-25T19:44:08.015661+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T19:44:08.021019+0000","last_undegraded":"2024-04-25T19:44:08.021019+0000","last_fullsized":"2024-04-25T19:44:08.021019+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":54,"acting":54,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672965,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":139264,"data_stored":43141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672965,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27740,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":733184,"data_stored":633509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27740,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":733184,"data_stored":633509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31316,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340524,"statfs":{"total":96636764160,"available":96604696576,"internally_reserved":0,"allocated":139264,"data_stored":43141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T19:44:14.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T19:44:14.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.242+0000 7f78215a4640 1 Processor -- start 2024-04-25T19:44:14.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.242+0000 7f78215a4640 1 -- start start 2024-04-25T19:44:14.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.243+0000 7f78215a4640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f781c06f1a0 0x7f781c145fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:14.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.243+0000 7f78215a4640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f781c146520 0x7f781c1513f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:14.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.243+0000 7f78215a4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f781c0ac590 con 0x7f781c0aabc0 2024-04-25T19:44:14.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.244+0000 7f78215a4640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f781c0ab100 con 0x7f781c06f1a0 2024-04-25T19:44:14.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.244+0000 7f78215a4640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f781c0ab590 con 0x7f781c146520 2024-04-25T19:44:14.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.244+0000 7f781b7fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f781c146520 0x7f781c1513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:14.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.244+0000 7f781affd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f781c0aabc0 0x7f781c0aba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37996/0 (socket says 172.21.15.29:37996) 2024-04-25T19:44:14.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.244+0000 7f781affd640 1 -- 172.21.15.29:0/1677427588 learned_addr learned my addr 172.21.15.29:0/1677427588 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:14.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.244+0000 7f781a7fc640 1 --2- 172.21.15.29:0/1677427588 >> v2:172.21.15.102:3300/0 conn(0x7f781c06f1a0 0x7f781c145fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:14.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.244+0000 7f7819ffb640 1 -- 172.21.15.29:0/1677427588 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3866827861 0 0) 0x7f781c0ac590 con 0x7f781c0aabc0 2024-04-25T19:44:14.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.244+0000 7f781b7fe640 1 -- 172.21.15.29:0/1677427588 >> v2:172.21.15.102:3300/0 conn(0x7f781c06f1a0 msgr2=0x7f781c145fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:14.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.244+0000 7f781b7fe640 1 --2- 172.21.15.29:0/1677427588 >> v2:172.21.15.102:3300/0 conn(0x7f781c06f1a0 0x7f781c145fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.244+0000 7f781b7fe640 1 -- 172.21.15.29:0/1677427588 >> v1:172.21.15.29:6789/0 conn(0x7f781c0aabc0 legacy=0x7f781c0aba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:14.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.244+0000 7f781b7fe640 1 -- 172.21.15.29:0/1677427588 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f781c0ac1a0 con 0x7f781c146520 2024-04-25T19:44:14.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.245+0000 7f781b7fe640 1 --2- 172.21.15.29:0/1677427588 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f781c146520 0x7f781c1513f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f78140097a0 tx=0x7f781402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a2ab36e8a2a17fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:14.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.245+0000 7f7819ffb640 1 -- 172.21.15.29:0/1677427588 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7814035020 con 0x7f781c146520 2024-04-25T19:44:14.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.245+0000 7f7819ffb640 1 -- 172.21.15.29:0/1677427588 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7814031830 con 0x7f781c146520 2024-04-25T19:44:14.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.245+0000 7f7819ffb640 1 -- 172.21.15.29:0/1677427588 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7814031c00 con 0x7f781c146520 2024-04-25T19:44:14.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.246+0000 7f78215a4640 1 -- 172.21.15.29:0/1677427588 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f781c146520 msgr2=0x7f781c1513f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:14.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.246+0000 7f78215a4640 1 --2- 172.21.15.29:0/1677427588 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f781c146520 0x7f781c1513f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f78140097a0 tx=0x7f781402b8d0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.246+0000 7f78215a4640 1 -- 172.21.15.29:0/1677427588 shutdown_connections 2024-04-25T19:44:14.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.246+0000 7f78215a4640 1 --2- 172.21.15.29:0/1677427588 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f781c146520 0x7f781c1513f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.246+0000 7f78215a4640 1 --2- 172.21.15.29:0/1677427588 >> v2:172.21.15.102:3300/0 conn(0x7f781c06f1a0 0x7f781c145fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.246+0000 7f78215a4640 1 -- 172.21.15.29:0/1677427588 >> 172.21.15.29:0/1677427588 conn(0x7f781c072b90 msgr2=0x7f781c0a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:14.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.246+0000 7f78215a4640 1 -- 172.21.15.29:0/1677427588 shutdown_connections 2024-04-25T19:44:14.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.246+0000 7f78215a4640 1 -- 172.21.15.29:0/1677427588 wait complete. 2024-04-25T19:44:14.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.247+0000 7f78215a4640 1 Processor -- start 2024-04-25T19:44:14.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.247+0000 7f78215a4640 1 -- start start 2024-04-25T19:44:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.248+0000 7f78215a4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f781c06f1a0 0x7f781c1400a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.248+0000 7f78215a4640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f781c0aabc0 0x7f781c0b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.248+0000 7f78215a4640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f781c146520 0x7f781c132510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.248+0000 7f78215a4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f781c151ab0 con 0x7f781c06f1a0 2024-04-25T19:44:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.248+0000 7f78215a4640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f781c151930 con 0x7f781c146520 2024-04-25T19:44:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.248+0000 7f78215a4640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f781c151c30 con 0x7f781c0aabc0 2024-04-25T19:44:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.248+0000 7f781affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f781c06f1a0 0x7f781c1400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.248+0000 7f781a7fc640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f781c0aabc0 0x7f781c0b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.248+0000 7f781a7fc640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f781c0aabc0 0x7f781c0b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34434/0 (socket says 172.21.15.29:34434) 2024-04-25T19:44:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.248+0000 7f781affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f781c06f1a0 0x7f781c1400a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36500/0 (socket says 172.21.15.29:36500) 2024-04-25T19:44:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.248+0000 7f781a7fc640 1 -- 172.21.15.29:0/3048174080 learned_addr learned my addr 172.21.15.29:0/3048174080 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.249+0000 7f781b7fe640 1 --2- 172.21.15.29:0/3048174080 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f781c146520 0x7f781c132510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.249+0000 7f781a7fc640 1 -- 172.21.15.29:0/3048174080 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f781c146520 msgr2=0x7f781c132510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.249+0000 7f781a7fc640 1 --2- 172.21.15.29:0/3048174080 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f781c146520 0x7f781c132510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.249+0000 7f781a7fc640 1 -- 172.21.15.29:0/3048174080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f781c06f1a0 msgr2=0x7f781c1400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:14.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.249+0000 7f781a7fc640 1 --2- 172.21.15.29:0/3048174080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f781c06f1a0 0x7f781c1400a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.249+0000 7f781a7fc640 1 -- 172.21.15.29:0/3048174080 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f781c133230 con 0x7f781c0aabc0 2024-04-25T19:44:14.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.249+0000 7f781affd640 1 --2- 172.21.15.29:0/3048174080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f781c06f1a0 0x7f781c1400a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:14.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.249+0000 7f781a7fc640 1 --2- 172.21.15.29:0/3048174080 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f781c0aabc0 0x7f781c0b1d50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f780400d5f0 tx=0x7f780400dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:14.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.249+0000 7f78137fe640 1 -- 172.21.15.29:0/3048174080 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78040125f0 con 0x7f781c0aabc0 2024-04-25T19:44:14.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.249+0000 7f78137fe640 1 -- 172.21.15.29:0/3048174080 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7804019070 con 0x7f781c0aabc0 2024-04-25T19:44:14.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.249+0000 7f78215a4640 1 -- 172.21.15.29:0/3048174080 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f781c132a50 con 0x7f781c0aabc0 2024-04-25T19:44:14.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.249+0000 7f78137fe640 1 -- 172.21.15.29:0/3048174080 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78040129d0 con 0x7f781c0aabc0 2024-04-25T19:44:14.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.249+0000 7f78215a4640 1 -- 172.21.15.29:0/3048174080 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f781c1362c0 con 0x7f781c0aabc0 2024-04-25T19:44:14.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.250+0000 7f78215a4640 1 -- 172.21.15.29:0/3048174080 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f781c0507b0 con 0x7f781c0aabc0 2024-04-25T19:44:14.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.252+0000 7f78137fe640 1 -- 172.21.15.29:0/3048174080 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f7804012b70 con 0x7f781c0aabc0 2024-04-25T19:44:14.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.252+0000 7f78137fe640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:14.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.253+0000 7f78137fe640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:14.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.253+0000 7f78137fe640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:14.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.253+0000 7f78137fe640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:14.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.253+0000 7f78137fe640 1 --2- 172.21.15.29:0/3048174080 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f77fc0ac530 0x7f77fc0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:14.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.253+0000 7f78137fe640 1 -- 172.21.15.29:0/3048174080 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f78040da220 con 0x7f781c0aabc0 2024-04-25T19:44:14.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.253+0000 7f781affd640 1 --2- 172.21.15.29:0/3048174080 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f77fc0ac530 0x7f77fc0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:14.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.254+0000 7f781affd640 1 --2- 172.21.15.29:0/3048174080 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f77fc0ac530 0x7f77fc0aea40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f780c0012e0 tx=0x7f780c001630 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:14.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.256+0000 7f78137fe640 1 -- 172.21.15.29:0/3048174080 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f78040a4e90 con 0x7f781c0aabc0 2024-04-25T19:44:14.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.438+0000 7f78215a4640 20 mgrc start_command cmd: [{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}] 2024-04-25T19:44:14.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.438+0000 7f78215a4640 1 -- 172.21.15.29:0/3048174080 --> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f781c06db20 con 0x7f77fc0ac530 2024-04-25T19:44:14.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.445+0000 7f78137fe640 1 -- 172.21.15.29:0/3048174080 <== mgr.4108 v2:172.21.15.102:6824/1439704316 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+268585 (secure 0 0 0) 0x7f781c06db20 con 0x7f77fc0ac530 2024-04-25T19:44:14.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.445+0000 7f78137fe640 20 mgrc handle_command_reply tid 0 r 0 2024-04-25T19:44:14.449 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:14.450 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-04-25T19:44:14.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.448+0000 7f78215a4640 10 mgrc shutdown 2024-04-25T19:44:14.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.448+0000 7f78215a4640 1 -- 172.21.15.29:0/3048174080 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f77fc0ac530 msgr2=0x7f77fc0aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:14.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.448+0000 7f78215a4640 1 --2- 172.21.15.29:0/3048174080 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f77fc0ac530 0x7f77fc0aea40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f780c0012e0 tx=0x7f780c001630 comp rx=0 tx=0).stop 2024-04-25T19:44:14.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.448+0000 7f78215a4640 1 -- 172.21.15.29:0/3048174080 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f781c0aabc0 msgr2=0x7f781c0b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:14.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.448+0000 7f78215a4640 1 --2- 172.21.15.29:0/3048174080 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f781c0aabc0 0x7f781c0b1d50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f780400d5f0 tx=0x7f780400dab0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.448+0000 7f78215a4640 1 -- 172.21.15.29:0/3048174080 shutdown_connections 2024-04-25T19:44:14.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.448+0000 7f78215a4640 1 --2- 172.21.15.29:0/3048174080 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f77fc0ac530 0x7f77fc0aea40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.448+0000 7f78215a4640 1 --2- 172.21.15.29:0/3048174080 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f781c146520 0x7f781c132510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.448+0000 7f78215a4640 1 --2- 172.21.15.29:0/3048174080 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f781c0aabc0 0x7f781c0b1d50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.448+0000 7f78215a4640 1 --2- 172.21.15.29:0/3048174080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f781c06f1a0 0x7f781c1400a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.448+0000 7f78215a4640 1 -- 172.21.15.29:0/3048174080 >> 172.21.15.29:0/3048174080 conn(0x7f781c072b90 msgr2=0x7f781c06cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:14.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.448+0000 7f78215a4640 1 -- 172.21.15.29:0/3048174080 shutdown_connections 2024-04-25T19:44:14.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.448+0000 7f78215a4640 1 -- 172.21.15.29:0/3048174080 wait complete. 2024-04-25T19:44:14.460 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-04-25T19:44:14.082910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":114192,"kb_used_data":1944,"kb_used_omap":6,"kb_used_meta":112185,"kb_avail":377373168,"statfs":{"total":386547056640,"available":386430124032,"internally_reserved":0,"allocated":1990656,"data_stored":1436532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":114878230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001229"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043136+0000","last_change":"2024-04-25T19:44:10.065844+0000","last_active":"2024-04-25T19:44:11.043136+0000","last_peered":"2024-04-25T19:44:11.043136+0000","last_clean":"2024-04-25T19:44:11.043136+0000","last_became_active":"2024-04-25T19:44:10.065543+0000","last_became_peered":"2024-04-25T19:44:10.065543+0000","last_unstale":"2024-04-25T19:44:11.043136+0000","last_undegraded":"2024-04-25T19:44:11.043136+0000","last_fullsized":"2024-04-25T19:44:11.043136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:28.992782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042952+0000","last_change":"2024-04-25T19:44:10.054974+0000","last_active":"2024-04-25T19:44:11.042952+0000","last_peered":"2024-04-25T19:44:11.042952+0000","last_clean":"2024-04-25T19:44:11.042952+0000","last_became_active":"2024-04-25T19:44:10.054042+0000","last_became_peered":"2024-04-25T19:44:10.054042+0000","last_unstale":"2024-04-25T19:44:11.042952+0000","last_undegraded":"2024-04-25T19:44:11.042952+0000","last_fullsized":"2024-04-25T19:44:11.042952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:56.154645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436639+0000","last_change":"2024-04-25T19:44:10.060234+0000","last_active":"2024-04-25T19:44:11.436639+0000","last_peered":"2024-04-25T19:44:11.436639+0000","last_clean":"2024-04-25T19:44:11.436639+0000","last_became_active":"2024-04-25T19:44:10.059094+0000","last_became_peered":"2024-04-25T19:44:10.059094+0000","last_unstale":"2024-04-25T19:44:11.436639+0000","last_undegraded":"2024-04-25T19:44:11.436639+0000","last_fullsized":"2024-04-25T19:44:11.436639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:18.383491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745013+0000","last_change":"2024-04-25T19:44:10.056978+0000","last_active":"2024-04-25T19:44:11.745013+0000","last_peered":"2024-04-25T19:44:11.745013+0000","last_clean":"2024-04-25T19:44:11.745013+0000","last_became_active":"2024-04-25T19:44:10.051835+0000","last_became_peered":"2024-04-25T19:44:10.051835+0000","last_unstale":"2024-04-25T19:44:11.745013+0000","last_undegraded":"2024-04-25T19:44:11.745013+0000","last_fullsized":"2024-04-25T19:44:11.745013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:05:36.546123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436228+0000","last_change":"2024-04-25T19:44:10.058920+0000","last_active":"2024-04-25T19:44:11.436228+0000","last_peered":"2024-04-25T19:44:11.436228+0000","last_clean":"2024-04-25T19:44:11.436228+0000","last_became_active":"2024-04-25T19:44:10.058585+0000","last_became_peered":"2024-04-25T19:44:10.058585+0000","last_unstale":"2024-04-25T19:44:11.436228+0000","last_undegraded":"2024-04-25T19:44:11.436228+0000","last_fullsized":"2024-04-25T19:44:11.436228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:48:06.055133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435841+0000","last_change":"2024-04-25T19:44:10.055592+0000","last_active":"2024-04-25T19:44:11.435841+0000","last_peered":"2024-04-25T19:44:11.435841+0000","last_clean":"2024-04-25T19:44:11.435841+0000","last_became_active":"2024-04-25T19:44:10.053926+0000","last_became_peered":"2024-04-25T19:44:10.053926+0000","last_unstale":"2024-04-25T19:44:11.435841+0000","last_undegraded":"2024-04-25T19:44:11.435841+0000","last_fullsized":"2024-04-25T19:44:11.435841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:01.889093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435699+0000","last_change":"2024-04-25T19:44:10.058739+0000","last_active":"2024-04-25T19:44:11.435699+0000","last_peered":"2024-04-25T19:44:11.435699+0000","last_clean":"2024-04-25T19:44:11.435699+0000","last_became_active":"2024-04-25T19:44:10.053725+0000","last_became_peered":"2024-04-25T19:44:10.053725+0000","last_unstale":"2024-04-25T19:44:11.435699+0000","last_undegraded":"2024-04-25T19:44:11.435699+0000","last_fullsized":"2024-04-25T19:44:11.435699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:49.082468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745508+0000","last_change":"2024-04-25T19:44:10.069350+0000","last_active":"2024-04-25T19:44:11.745508+0000","last_peered":"2024-04-25T19:44:11.745508+0000","last_clean":"2024-04-25T19:44:11.745508+0000","last_became_active":"2024-04-25T19:44:10.069065+0000","last_became_peered":"2024-04-25T19:44:10.069065+0000","last_unstale":"2024-04-25T19:44:11.745508+0000","last_undegraded":"2024-04-25T19:44:11.745508+0000","last_fullsized":"2024-04-25T19:44:11.745508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:43:06.731936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042767+0000","last_change":"2024-04-25T19:44:10.052909+0000","last_active":"2024-04-25T19:44:11.042767+0000","last_peered":"2024-04-25T19:44:11.042767+0000","last_clean":"2024-04-25T19:44:11.042767+0000","last_became_active":"2024-04-25T19:44:10.052592+0000","last_became_peered":"2024-04-25T19:44:10.052592+0000","last_unstale":"2024-04-25T19:44:11.042767+0000","last_undegraded":"2024-04-25T19:44:11.042767+0000","last_fullsized":"2024-04-25T19:44:11.042767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:50.564937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745396+0000","last_change":"2024-04-25T19:44:10.060703+0000","last_active":"2024-04-25T19:44:11.745396+0000","last_peered":"2024-04-25T19:44:11.745396+0000","last_clean":"2024-04-25T19:44:11.745396+0000","last_became_active":"2024-04-25T19:44:10.058013+0000","last_became_peered":"2024-04-25T19:44:10.058013+0000","last_unstale":"2024-04-25T19:44:11.745396+0000","last_undegraded":"2024-04-25T19:44:11.745396+0000","last_fullsized":"2024-04-25T19:44:11.745396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:15:17.871931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745517+0000","last_change":"2024-04-25T19:44:10.049572+0000","last_active":"2024-04-25T19:44:11.745517+0000","last_peered":"2024-04-25T19:44:11.745517+0000","last_clean":"2024-04-25T19:44:11.745517+0000","last_became_active":"2024-04-25T19:44:10.049249+0000","last_became_peered":"2024-04-25T19:44:10.049249+0000","last_unstale":"2024-04-25T19:44:11.745517+0000","last_undegraded":"2024-04-25T19:44:11.745517+0000","last_fullsized":"2024-04-25T19:44:11.745517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:52:09.103201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043197+0000","last_change":"2024-04-25T19:44:10.056274+0000","last_active":"2024-04-25T19:44:11.043197+0000","last_peered":"2024-04-25T19:44:11.043197+0000","last_clean":"2024-04-25T19:44:11.043197+0000","last_became_active":"2024-04-25T19:44:10.055981+0000","last_became_peered":"2024-04-25T19:44:10.055981+0000","last_unstale":"2024-04-25T19:44:11.043197+0000","last_undegraded":"2024-04-25T19:44:11.043197+0000","last_fullsized":"2024-04-25T19:44:11.043197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:52:52.432869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042230+0000","last_change":"2024-04-25T19:44:10.044363+0000","last_active":"2024-04-25T19:44:11.042230+0000","last_peered":"2024-04-25T19:44:11.042230+0000","last_clean":"2024-04-25T19:44:11.042230+0000","last_became_active":"2024-04-25T19:44:10.044033+0000","last_became_peered":"2024-04-25T19:44:10.044033+0000","last_unstale":"2024-04-25T19:44:11.042230+0000","last_undegraded":"2024-04-25T19:44:11.042230+0000","last_fullsized":"2024-04-25T19:44:11.042230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:19.654106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.744782+0000","last_change":"2024-04-25T19:44:10.068850+0000","last_active":"2024-04-25T19:44:11.744782+0000","last_peered":"2024-04-25T19:44:11.744782+0000","last_clean":"2024-04-25T19:44:11.744782+0000","last_became_active":"2024-04-25T19:44:10.068565+0000","last_became_peered":"2024-04-25T19:44:10.068565+0000","last_unstale":"2024-04-25T19:44:11.744782+0000","last_undegraded":"2024-04-25T19:44:11.744782+0000","last_fullsized":"2024-04-25T19:44:11.744782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:40.201502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.744781+0000","last_change":"2024-04-25T19:44:10.056683+0000","last_active":"2024-04-25T19:44:11.744781+0000","last_peered":"2024-04-25T19:44:11.744781+0000","last_clean":"2024-04-25T19:44:11.744781+0000","last_became_active":"2024-04-25T19:44:10.056375+0000","last_became_peered":"2024-04-25T19:44:10.056375+0000","last_unstale":"2024-04-25T19:44:11.744781+0000","last_undegraded":"2024-04-25T19:44:11.744781+0000","last_fullsized":"2024-04-25T19:44:11.744781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:50:21.528308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435257+0000","last_change":"2024-04-25T19:44:10.056384+0000","last_active":"2024-04-25T19:44:11.435257+0000","last_peered":"2024-04-25T19:44:11.435257+0000","last_clean":"2024-04-25T19:44:11.435257+0000","last_became_active":"2024-04-25T19:44:10.054724+0000","last_became_peered":"2024-04-25T19:44:10.054724+0000","last_unstale":"2024-04-25T19:44:11.435257+0000","last_undegraded":"2024-04-25T19:44:11.435257+0000","last_fullsized":"2024-04-25T19:44:11.435257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:46:59.212591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745764+0000","last_change":"2024-04-25T19:44:10.069348+0000","last_active":"2024-04-25T19:44:11.745764+0000","last_peered":"2024-04-25T19:44:11.745764+0000","last_clean":"2024-04-25T19:44:11.745764+0000","last_became_active":"2024-04-25T19:44:10.068699+0000","last_became_peered":"2024-04-25T19:44:10.068699+0000","last_unstale":"2024-04-25T19:44:11.745764+0000","last_undegraded":"2024-04-25T19:44:11.745764+0000","last_fullsized":"2024-04-25T19:44:11.745764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:33:58.495931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436501+0000","last_change":"2024-04-25T19:44:10.061949+0000","last_active":"2024-04-25T19:44:11.436501+0000","last_peered":"2024-04-25T19:44:11.436501+0000","last_clean":"2024-04-25T19:44:11.436501+0000","last_became_active":"2024-04-25T19:44:10.061187+0000","last_became_peered":"2024-04-25T19:44:10.061187+0000","last_unstale":"2024-04-25T19:44:11.436501+0000","last_undegraded":"2024-04-25T19:44:11.436501+0000","last_fullsized":"2024-04-25T19:44:11.436501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:41:07.670796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042266+0000","last_change":"2024-04-25T19:44:10.045506+0000","last_active":"2024-04-25T19:44:11.042266+0000","last_peered":"2024-04-25T19:44:11.042266+0000","last_clean":"2024-04-25T19:44:11.042266+0000","last_became_active":"2024-04-25T19:44:10.044359+0000","last_became_peered":"2024-04-25T19:44:10.044359+0000","last_unstale":"2024-04-25T19:44:11.042266+0000","last_undegraded":"2024-04-25T19:44:11.042266+0000","last_fullsized":"2024-04-25T19:44:11.042266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:43:23.290058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745208+0000","last_change":"2024-04-25T19:44:10.051944+0000","last_active":"2024-04-25T19:44:11.745208+0000","last_peered":"2024-04-25T19:44:11.745208+0000","last_clean":"2024-04-25T19:44:11.745208+0000","last_became_active":"2024-04-25T19:44:10.051469+0000","last_became_peered":"2024-04-25T19:44:10.051469+0000","last_unstale":"2024-04-25T19:44:11.745208+0000","last_undegraded":"2024-04-25T19:44:11.745208+0000","last_fullsized":"2024-04-25T19:44:11.745208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:46:54.526883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745057+0000","last_change":"2024-04-25T19:44:10.060425+0000","last_active":"2024-04-25T19:44:11.745057+0000","last_peered":"2024-04-25T19:44:11.745057+0000","last_clean":"2024-04-25T19:44:11.745057+0000","last_became_active":"2024-04-25T19:44:10.059664+0000","last_became_peered":"2024-04-25T19:44:10.059664+0000","last_unstale":"2024-04-25T19:44:11.745057+0000","last_undegraded":"2024-04-25T19:44:11.745057+0000","last_fullsized":"2024-04-25T19:44:11.745057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:10:11.581750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.744779+0000","last_change":"2024-04-25T19:44:10.068792+0000","last_active":"2024-04-25T19:44:11.744779+0000","last_peered":"2024-04-25T19:44:11.744779+0000","last_clean":"2024-04-25T19:44:11.744779+0000","last_became_active":"2024-04-25T19:44:10.068477+0000","last_became_peered":"2024-04-25T19:44:10.068477+0000","last_unstale":"2024-04-25T19:44:11.744779+0000","last_undegraded":"2024-04-25T19:44:11.744779+0000","last_fullsized":"2024-04-25T19:44:11.744779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:18.399916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435517+0000","last_change":"2024-04-25T19:44:10.058966+0000","last_active":"2024-04-25T19:44:11.435517+0000","last_peered":"2024-04-25T19:44:11.435517+0000","last_clean":"2024-04-25T19:44:11.435517+0000","last_became_active":"2024-04-25T19:44:10.058083+0000","last_became_peered":"2024-04-25T19:44:10.058083+0000","last_unstale":"2024-04-25T19:44:11.435517+0000","last_undegraded":"2024-04-25T19:44:11.435517+0000","last_fullsized":"2024-04-25T19:44:11.435517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:34.810052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745205+0000","last_change":"2024-04-25T19:44:10.057670+0000","last_active":"2024-04-25T19:44:11.745205+0000","last_peered":"2024-04-25T19:44:11.745205+0000","last_clean":"2024-04-25T19:44:11.745205+0000","last_became_active":"2024-04-25T19:44:10.057360+0000","last_became_peered":"2024-04-25T19:44:10.057360+0000","last_unstale":"2024-04-25T19:44:11.745205+0000","last_undegraded":"2024-04-25T19:44:11.745205+0000","last_fullsized":"2024-04-25T19:44:11.745205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:18:02.918434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745808+0000","last_change":"2024-04-25T19:44:10.069281+0000","last_active":"2024-04-25T19:44:11.745808+0000","last_peered":"2024-04-25T19:44:11.745808+0000","last_clean":"2024-04-25T19:44:11.745808+0000","last_became_active":"2024-04-25T19:44:10.068327+0000","last_became_peered":"2024-04-25T19:44:10.068327+0000","last_unstale":"2024-04-25T19:44:11.745808+0000","last_undegraded":"2024-04-25T19:44:11.745808+0000","last_fullsized":"2024-04-25T19:44:11.745808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:46:59.212591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042716+0000","last_change":"2024-04-25T19:44:10.055433+0000","last_active":"2024-04-25T19:44:11.042716+0000","last_peered":"2024-04-25T19:44:11.042716+0000","last_clean":"2024-04-25T19:44:11.042716+0000","last_became_active":"2024-04-25T19:44:10.055058+0000","last_became_peered":"2024-04-25T19:44:10.055058+0000","last_unstale":"2024-04-25T19:44:11.042716+0000","last_undegraded":"2024-04-25T19:44:11.042716+0000","last_fullsized":"2024-04-25T19:44:11.042716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:56.154645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.437038+0000","last_change":"2024-04-25T19:44:10.064610+0000","last_active":"2024-04-25T19:44:11.437038+0000","last_peered":"2024-04-25T19:44:11.437038+0000","last_clean":"2024-04-25T19:44:11.437038+0000","last_became_active":"2024-04-25T19:44:10.064295+0000","last_became_peered":"2024-04-25T19:44:10.064295+0000","last_unstale":"2024-04-25T19:44:11.437038+0000","last_undegraded":"2024-04-25T19:44:11.437038+0000","last_fullsized":"2024-04-25T19:44:11.437038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:09.492760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043265+0000","last_change":"2024-04-25T19:44:10.050485+0000","last_active":"2024-04-25T19:44:11.043265+0000","last_peered":"2024-04-25T19:44:11.043265+0000","last_clean":"2024-04-25T19:44:11.043265+0000","last_became_active":"2024-04-25T19:44:10.050141+0000","last_became_peered":"2024-04-25T19:44:10.050141+0000","last_unstale":"2024-04-25T19:44:11.043265+0000","last_undegraded":"2024-04-25T19:44:11.043265+0000","last_fullsized":"2024-04-25T19:44:11.043265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:40.201502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042339+0000","last_change":"2024-04-25T19:44:10.044738+0000","last_active":"2024-04-25T19:44:11.042339+0000","last_peered":"2024-04-25T19:44:11.042339+0000","last_clean":"2024-04-25T19:44:11.042339+0000","last_became_active":"2024-04-25T19:44:10.044326+0000","last_became_peered":"2024-04-25T19:44:10.044326+0000","last_unstale":"2024-04-25T19:44:11.042339+0000","last_undegraded":"2024-04-25T19:44:11.042339+0000","last_fullsized":"2024-04-25T19:44:11.042339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:21.909648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745442+0000","last_change":"2024-04-25T19:44:10.062939+0000","last_active":"2024-04-25T19:44:11.745442+0000","last_peered":"2024-04-25T19:44:11.745442+0000","last_clean":"2024-04-25T19:44:11.745442+0000","last_became_active":"2024-04-25T19:44:10.057793+0000","last_became_peered":"2024-04-25T19:44:10.057793+0000","last_unstale":"2024-04-25T19:44:11.745442+0000","last_undegraded":"2024-04-25T19:44:11.745442+0000","last_fullsized":"2024-04-25T19:44:11.745442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:30.311117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.041761+0000","last_change":"2024-04-25T19:44:10.053059+0000","last_active":"2024-04-25T19:44:11.041761+0000","last_peered":"2024-04-25T19:44:11.041761+0000","last_clean":"2024-04-25T19:44:11.041761+0000","last_became_active":"2024-04-25T19:44:10.052494+0000","last_became_peered":"2024-04-25T19:44:10.052494+0000","last_unstale":"2024-04-25T19:44:11.041761+0000","last_undegraded":"2024-04-25T19:44:11.041761+0000","last_fullsized":"2024-04-25T19:44:11.041761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:38:53.994523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745521+0000","last_change":"2024-04-25T19:44:10.060010+0000","last_active":"2024-04-25T19:44:11.745521+0000","last_peered":"2024-04-25T19:44:11.745521+0000","last_clean":"2024-04-25T19:44:11.745521+0000","last_became_active":"2024-04-25T19:44:10.059598+0000","last_became_peered":"2024-04-25T19:44:10.059598+0000","last_unstale":"2024-04-25T19:44:11.745521+0000","last_undegraded":"2024-04-25T19:44:11.745521+0000","last_fullsized":"2024-04-25T19:44:11.745521+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:40.247386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435938+0000","last_change":"2024-04-25T19:44:10.059147+0000","last_active":"2024-04-25T19:44:11.435938+0000","last_peered":"2024-04-25T19:44:11.435938+0000","last_clean":"2024-04-25T19:44:11.435938+0000","last_became_active":"2024-04-25T19:44:10.058363+0000","last_became_peered":"2024-04-25T19:44:10.058363+0000","last_unstale":"2024-04-25T19:44:11.435938+0000","last_undegraded":"2024-04-25T19:44:11.435938+0000","last_fullsized":"2024-04-25T19:44:11.435938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:05:57.081479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436415+0000","last_change":"2024-04-25T19:44:09.056791+0000","last_active":"2024-04-25T19:44:11.436415+0000","last_peered":"2024-04-25T19:44:11.436415+0000","last_clean":"2024-04-25T19:44:11.436415+0000","last_became_active":"2024-04-25T19:44:09.056430+0000","last_became_peered":"2024-04-25T19:44:09.056430+0000","last_unstale":"2024-04-25T19:44:11.436415+0000","last_undegraded":"2024-04-25T19:44:11.436415+0000","last_fullsized":"2024-04-25T19:44:11.436415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:18.652125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436777+0000","last_change":"2024-04-25T19:44:10.062414+0000","last_active":"2024-04-25T19:44:11.436777+0000","last_peered":"2024-04-25T19:44:11.436777+0000","last_clean":"2024-04-25T19:44:11.436777+0000","last_became_active":"2024-04-25T19:44:10.061691+0000","last_became_peered":"2024-04-25T19:44:10.061691+0000","last_unstale":"2024-04-25T19:44:11.436777+0000","last_undegraded":"2024-04-25T19:44:11.436777+0000","last_fullsized":"2024-04-25T19:44:11.436777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:29:27.722435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043208+0000","last_change":"2024-04-25T19:44:09.052305+0000","last_active":"2024-04-25T19:44:11.043208+0000","last_peered":"2024-04-25T19:44:11.043208+0000","last_clean":"2024-04-25T19:44:11.043208+0000","last_became_active":"2024-04-25T19:44:09.051968+0000","last_became_peered":"2024-04-25T19:44:09.051968+0000","last_unstale":"2024-04-25T19:44:11.043208+0000","last_undegraded":"2024-04-25T19:44:11.043208+0000","last_fullsized":"2024-04-25T19:44:11.043208+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:11:22.034588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436495+0000","last_change":"2024-04-25T19:44:10.062199+0000","last_active":"2024-04-25T19:44:11.436495+0000","last_peered":"2024-04-25T19:44:11.436495+0000","last_clean":"2024-04-25T19:44:11.436495+0000","last_became_active":"2024-04-25T19:44:10.061613+0000","last_became_peered":"2024-04-25T19:44:10.061613+0000","last_unstale":"2024-04-25T19:44:11.436495+0000","last_undegraded":"2024-04-25T19:44:11.436495+0000","last_fullsized":"2024-04-25T19:44:11.436495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:56:49.470340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436342+0000","last_change":"2024-04-25T19:44:09.058706+0000","last_active":"2024-04-25T19:44:11.436342+0000","last_peered":"2024-04-25T19:44:11.436342+0000","last_clean":"2024-04-25T19:44:11.436342+0000","last_became_active":"2024-04-25T19:44:09.057009+0000","last_became_peered":"2024-04-25T19:44:09.057009+0000","last_unstale":"2024-04-25T19:44:11.436342+0000","last_undegraded":"2024-04-25T19:44:11.436342+0000","last_fullsized":"2024-04-25T19:44:11.436342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:17:26.264206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043125+0000","last_change":"2024-04-25T19:44:10.058621+0000","last_active":"2024-04-25T19:44:11.043125+0000","last_peered":"2024-04-25T19:44:11.043125+0000","last_clean":"2024-04-25T19:44:11.043125+0000","last_became_active":"2024-04-25T19:44:10.058237+0000","last_became_peered":"2024-04-25T19:44:10.058237+0000","last_unstale":"2024-04-25T19:44:11.043125+0000","last_undegraded":"2024-04-25T19:44:11.043125+0000","last_fullsized":"2024-04-25T19:44:11.043125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:18.399916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745612+0000","last_change":"2024-04-25T19:44:09.054405+0000","last_active":"2024-04-25T19:44:11.745612+0000","last_peered":"2024-04-25T19:44:11.745612+0000","last_clean":"2024-04-25T19:44:11.745612+0000","last_became_active":"2024-04-25T19:44:09.054065+0000","last_became_peered":"2024-04-25T19:44:09.054065+0000","last_unstale":"2024-04-25T19:44:11.745612+0000","last_undegraded":"2024-04-25T19:44:11.745612+0000","last_fullsized":"2024-04-25T19:44:11.745612+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:59.352345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043210+0000","last_change":"2024-04-25T19:44:09.047923+0000","last_active":"2024-04-25T19:44:11.043210+0000","last_peered":"2024-04-25T19:44:11.043210+0000","last_clean":"2024-04-25T19:44:11.043210+0000","last_became_active":"2024-04-25T19:44:09.047605+0000","last_became_peered":"2024-04-25T19:44:09.047605+0000","last_unstale":"2024-04-25T19:44:11.043210+0000","last_undegraded":"2024-04-25T19:44:11.043210+0000","last_fullsized":"2024-04-25T19:44:11.043210+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:30:43.807694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436073+0000","last_change":"2024-04-25T19:44:10.067620+0000","last_active":"2024-04-25T19:44:11.436073+0000","last_peered":"2024-04-25T19:44:11.436073+0000","last_clean":"2024-04-25T19:44:11.436073+0000","last_became_active":"2024-04-25T19:44:10.066364+0000","last_became_peered":"2024-04-25T19:44:10.066364+0000","last_unstale":"2024-04-25T19:44:11.436073+0000","last_undegraded":"2024-04-25T19:44:11.436073+0000","last_fullsized":"2024-04-25T19:44:11.436073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:20:30.629784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043379+0000","last_change":"2024-04-25T19:44:09.045871+0000","last_active":"2024-04-25T19:44:11.043379+0000","last_peered":"2024-04-25T19:44:11.043379+0000","last_clean":"2024-04-25T19:44:11.043379+0000","last_became_active":"2024-04-25T19:44:09.045017+0000","last_became_peered":"2024-04-25T19:44:09.045017+0000","last_unstale":"2024-04-25T19:44:11.043379+0000","last_undegraded":"2024-04-25T19:44:11.043379+0000","last_fullsized":"2024-04-25T19:44:11.043379+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:51:27.578108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.744790+0000","last_change":"2024-04-25T19:44:10.049695+0000","last_active":"2024-04-25T19:44:11.744790+0000","last_peered":"2024-04-25T19:44:11.744790+0000","last_clean":"2024-04-25T19:44:11.744790+0000","last_became_active":"2024-04-25T19:44:10.049361+0000","last_became_peered":"2024-04-25T19:44:10.049361+0000","last_unstale":"2024-04-25T19:44:11.744790+0000","last_undegraded":"2024-04-25T19:44:11.744790+0000","last_fullsized":"2024-04-25T19:44:11.744790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:57.562802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436867+0000","last_change":"2024-04-25T19:44:09.059742+0000","last_active":"2024-04-25T19:44:11.436867+0000","last_peered":"2024-04-25T19:44:11.436867+0000","last_clean":"2024-04-25T19:44:11.436867+0000","last_became_active":"2024-04-25T19:44:09.059318+0000","last_became_peered":"2024-04-25T19:44:09.059318+0000","last_unstale":"2024-04-25T19:44:11.436867+0000","last_undegraded":"2024-04-25T19:44:11.436867+0000","last_fullsized":"2024-04-25T19:44:11.436867+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:55.152664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043138+0000","last_change":"2024-04-25T19:44:10.056446+0000","last_active":"2024-04-25T19:44:11.043138+0000","last_peered":"2024-04-25T19:44:11.043138+0000","last_clean":"2024-04-25T19:44:11.043138+0000","last_became_active":"2024-04-25T19:44:10.056096+0000","last_became_peered":"2024-04-25T19:44:10.056096+0000","last_unstale":"2024-04-25T19:44:11.043138+0000","last_undegraded":"2024-04-25T19:44:11.043138+0000","last_fullsized":"2024-04-25T19:44:11.043138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:26:17.991276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436305+0000","last_change":"2024-04-25T19:44:09.054439+0000","last_active":"2024-04-25T19:44:11.436305+0000","last_peered":"2024-04-25T19:44:11.436305+0000","last_clean":"2024-04-25T19:44:11.436305+0000","last_became_active":"2024-04-25T19:44:09.053941+0000","last_became_peered":"2024-04-25T19:44:09.053941+0000","last_unstale":"2024-04-25T19:44:11.436305+0000","last_undegraded":"2024-04-25T19:44:11.436305+0000","last_fullsized":"2024-04-25T19:44:11.436305+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:55:44.329801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043164+0000","last_change":"2024-04-25T19:44:10.047455+0000","last_active":"2024-04-25T19:44:11.043164+0000","last_peered":"2024-04-25T19:44:11.043164+0000","last_clean":"2024-04-25T19:44:11.043164+0000","last_became_active":"2024-04-25T19:44:10.046766+0000","last_became_peered":"2024-04-25T19:44:10.046766+0000","last_unstale":"2024-04-25T19:44:11.043164+0000","last_undegraded":"2024-04-25T19:44:11.043164+0000","last_fullsized":"2024-04-25T19:44:11.043164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:46.058675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042351+0000","last_change":"2024-04-25T19:44:09.053423+0000","last_active":"2024-04-25T19:44:11.042351+0000","last_peered":"2024-04-25T19:44:11.042351+0000","last_clean":"2024-04-25T19:44:11.042351+0000","last_became_active":"2024-04-25T19:44:09.053138+0000","last_became_peered":"2024-04-25T19:44:09.053138+0000","last_unstale":"2024-04-25T19:44:11.042351+0000","last_undegraded":"2024-04-25T19:44:11.042351+0000","last_fullsized":"2024-04-25T19:44:11.042351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:11:22.034588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745419+0000","last_change":"2024-04-25T19:44:10.061890+0000","last_active":"2024-04-25T19:44:11.745419+0000","last_peered":"2024-04-25T19:44:11.745419+0000","last_clean":"2024-04-25T19:44:11.745419+0000","last_became_active":"2024-04-25T19:44:10.061525+0000","last_became_peered":"2024-04-25T19:44:10.061525+0000","last_unstale":"2024-04-25T19:44:11.745419+0000","last_undegraded":"2024-04-25T19:44:11.745419+0000","last_fullsized":"2024-04-25T19:44:11.745419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:33:29.948604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435794+0000","last_change":"2024-04-25T19:44:09.058395+0000","last_active":"2024-04-25T19:44:11.435794+0000","last_peered":"2024-04-25T19:44:11.435794+0000","last_clean":"2024-04-25T19:44:11.435794+0000","last_became_active":"2024-04-25T19:44:09.057411+0000","last_became_peered":"2024-04-25T19:44:09.057411+0000","last_unstale":"2024-04-25T19:44:11.435794+0000","last_undegraded":"2024-04-25T19:44:11.435794+0000","last_fullsized":"2024-04-25T19:44:11.435794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:53:44.798992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436721+0000","last_change":"2024-04-25T19:44:10.062688+0000","last_active":"2024-04-25T19:44:11.436721+0000","last_peered":"2024-04-25T19:44:11.436721+0000","last_clean":"2024-04-25T19:44:11.436721+0000","last_became_active":"2024-04-25T19:44:10.059124+0000","last_became_peered":"2024-04-25T19:44:10.059124+0000","last_unstale":"2024-04-25T19:44:11.436721+0000","last_undegraded":"2024-04-25T19:44:11.436721+0000","last_fullsized":"2024-04-25T19:44:11.436721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:18:06.928672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436491+0000","last_change":"2024-04-25T19:44:09.056965+0000","last_active":"2024-04-25T19:44:11.436491+0000","last_peered":"2024-04-25T19:44:11.436491+0000","last_clean":"2024-04-25T19:44:11.436491+0000","last_became_active":"2024-04-25T19:44:09.056557+0000","last_became_peered":"2024-04-25T19:44:09.056557+0000","last_unstale":"2024-04-25T19:44:11.436491+0000","last_undegraded":"2024-04-25T19:44:11.436491+0000","last_fullsized":"2024-04-25T19:44:11.436491+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:08:50.165109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043306+0000","last_change":"2024-04-25T19:44:10.066632+0000","last_active":"2024-04-25T19:44:11.043306+0000","last_peered":"2024-04-25T19:44:11.043306+0000","last_clean":"2024-04-25T19:44:11.043306+0000","last_became_active":"2024-04-25T19:44:10.066329+0000","last_became_peered":"2024-04-25T19:44:10.066329+0000","last_unstale":"2024-04-25T19:44:11.043306+0000","last_undegraded":"2024-04-25T19:44:11.043306+0000","last_fullsized":"2024-04-25T19:44:11.043306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:18.399916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435984+0000","last_change":"2024-04-25T19:44:09.058583+0000","last_active":"2024-04-25T19:44:11.435984+0000","last_peered":"2024-04-25T19:44:11.435984+0000","last_clean":"2024-04-25T19:44:11.435984+0000","last_became_active":"2024-04-25T19:44:09.057289+0000","last_became_peered":"2024-04-25T19:44:09.057289+0000","last_unstale":"2024-04-25T19:44:11.435984+0000","last_undegraded":"2024-04-25T19:44:11.435984+0000","last_fullsized":"2024-04-25T19:44:11.435984+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:43:22.288077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745913+0000","last_change":"2024-04-25T19:44:10.069026+0000","last_active":"2024-04-25T19:44:11.745913+0000","last_peered":"2024-04-25T19:44:11.745913+0000","last_clean":"2024-04-25T19:44:11.745913+0000","last_became_active":"2024-04-25T19:44:10.067900+0000","last_became_peered":"2024-04-25T19:44:10.067900+0000","last_unstale":"2024-04-25T19:44:11.745913+0000","last_undegraded":"2024-04-25T19:44:11.745913+0000","last_fullsized":"2024-04-25T19:44:11.745913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:51.252629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'72","reported_seq":116,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745679+0000","last_change":"2024-04-25T19:44:00.045570+0000","last_active":"2024-04-25T19:44:11.745679+0000","last_peered":"2024-04-25T19:44:11.745679+0000","last_clean":"2024-04-25T19:44:11.745679+0000","last_became_active":"2024-04-25T19:44:00.044414+0000","last_became_peered":"2024-04-25T19:44:00.044414+0000","last_unstale":"2024-04-25T19:44:11.745679+0000","last_undegraded":"2024-04-25T19:44:11.745679+0000","last_fullsized":"2024-04-25T19:44:11.745679+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:43:58.854910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:43:58.854910+0000","last_clean_scrub_stamp":"2024-04-25T19:43:58.854910+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:36.813186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436422+0000","last_change":"2024-04-25T19:44:04.907674+0000","last_active":"2024-04-25T19:44:11.436422+0000","last_peered":"2024-04-25T19:44:11.436422+0000","last_clean":"2024-04-25T19:44:11.436422+0000","last_became_active":"2024-04-25T19:44:04.907259+0000","last_became_peered":"2024-04-25T19:44:04.907259+0000","last_unstale":"2024-04-25T19:44:11.436422+0000","last_undegraded":"2024-04-25T19:44:11.436422+0000","last_fullsized":"2024-04-25T19:44:11.436422+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:58.023878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436111+0000","last_change":"2024-04-25T19:44:09.056946+0000","last_active":"2024-04-25T19:44:11.436111+0000","last_peered":"2024-04-25T19:44:11.436111+0000","last_clean":"2024-04-25T19:44:11.436111+0000","last_became_active":"2024-04-25T19:44:09.056544+0000","last_became_peered":"2024-04-25T19:44:09.056544+0000","last_unstale":"2024-04-25T19:44:11.436111+0000","last_undegraded":"2024-04-25T19:44:11.436111+0000","last_fullsized":"2024-04-25T19:44:11.436111+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:24:46.807744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436901+0000","last_change":"2024-04-25T19:44:10.060919+0000","last_active":"2024-04-25T19:44:11.436901+0000","last_peered":"2024-04-25T19:44:11.436901+0000","last_clean":"2024-04-25T19:44:11.436901+0000","last_became_active":"2024-04-25T19:44:10.059529+0000","last_became_peered":"2024-04-25T19:44:10.059529+0000","last_unstale":"2024-04-25T19:44:11.436901+0000","last_undegraded":"2024-04-25T19:44:11.436901+0000","last_fullsized":"2024-04-25T19:44:11.436901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:29.509488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436146+0000","last_change":"2024-04-25T19:44:04.904786+0000","last_active":"2024-04-25T19:44:11.436146+0000","last_peered":"2024-04-25T19:44:11.436146+0000","last_clean":"2024-04-25T19:44:11.436146+0000","last_became_active":"2024-04-25T19:44:04.904414+0000","last_became_peered":"2024-04-25T19:44:04.904414+0000","last_unstale":"2024-04-25T19:44:11.436146+0000","last_undegraded":"2024-04-25T19:44:11.436146+0000","last_fullsized":"2024-04-25T19:44:11.436146+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:48:36.719208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.041861+0000","last_change":"2024-04-25T19:44:09.040846+0000","last_active":"2024-04-25T19:44:11.041861+0000","last_peered":"2024-04-25T19:44:11.041861+0000","last_clean":"2024-04-25T19:44:11.041861+0000","last_became_active":"2024-04-25T19:44:09.040461+0000","last_became_peered":"2024-04-25T19:44:09.040461+0000","last_unstale":"2024-04-25T19:44:11.041861+0000","last_undegraded":"2024-04-25T19:44:11.041861+0000","last_fullsized":"2024-04-25T19:44:11.041861+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:48:47.178567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436718+0000","last_change":"2024-04-25T19:44:10.062165+0000","last_active":"2024-04-25T19:44:11.436718+0000","last_peered":"2024-04-25T19:44:11.436718+0000","last_clean":"2024-04-25T19:44:11.436718+0000","last_became_active":"2024-04-25T19:44:10.061626+0000","last_became_peered":"2024-04-25T19:44:10.061626+0000","last_unstale":"2024-04-25T19:44:11.436718+0000","last_undegraded":"2024-04-25T19:44:11.436718+0000","last_fullsized":"2024-04-25T19:44:11.436718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:03:01.981027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042646+0000","last_change":"2024-04-25T19:44:04.904405+0000","last_active":"2024-04-25T19:44:11.042646+0000","last_peered":"2024-04-25T19:44:11.042646+0000","last_clean":"2024-04-25T19:44:11.042646+0000","last_became_active":"2024-04-25T19:44:04.904045+0000","last_became_peered":"2024-04-25T19:44:04.904045+0000","last_unstale":"2024-04-25T19:44:11.042646+0000","last_undegraded":"2024-04-25T19:44:11.042646+0000","last_fullsized":"2024-04-25T19:44:11.042646+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:01:59.962699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.041684+0000","last_change":"2024-04-25T19:44:09.045036+0000","last_active":"2024-04-25T19:44:11.041684+0000","last_peered":"2024-04-25T19:44:11.041684+0000","last_clean":"2024-04-25T19:44:11.041684+0000","last_became_active":"2024-04-25T19:44:09.044686+0000","last_became_peered":"2024-04-25T19:44:09.044686+0000","last_unstale":"2024-04-25T19:44:11.041684+0000","last_undegraded":"2024-04-25T19:44:11.041684+0000","last_fullsized":"2024-04-25T19:44:11.041684+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:41:37.111120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042963+0000","last_change":"2024-04-25T19:44:10.053021+0000","last_active":"2024-04-25T19:44:11.042963+0000","last_peered":"2024-04-25T19:44:11.042963+0000","last_clean":"2024-04-25T19:44:11.042963+0000","last_became_active":"2024-04-25T19:44:10.052663+0000","last_became_peered":"2024-04-25T19:44:10.052663+0000","last_unstale":"2024-04-25T19:44:11.042963+0000","last_undegraded":"2024-04-25T19:44:11.042963+0000","last_fullsized":"2024-04-25T19:44:11.042963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:46.058675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436124+0000","last_change":"2024-04-25T19:44:04.907093+0000","last_active":"2024-04-25T19:44:11.436124+0000","last_peered":"2024-04-25T19:44:11.436124+0000","last_clean":"2024-04-25T19:44:11.436124+0000","last_became_active":"2024-04-25T19:44:04.906787+0000","last_became_peered":"2024-04-25T19:44:04.906787+0000","last_unstale":"2024-04-25T19:44:11.436124+0000","last_undegraded":"2024-04-25T19:44:11.436124+0000","last_fullsized":"2024-04-25T19:44:11.436124+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:28:59.374809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043241+0000","last_change":"2024-04-25T19:44:09.043484+0000","last_active":"2024-04-25T19:44:11.043241+0000","last_peered":"2024-04-25T19:44:11.043241+0000","last_clean":"2024-04-25T19:44:11.043241+0000","last_became_active":"2024-04-25T19:44:09.043155+0000","last_became_peered":"2024-04-25T19:44:09.043155+0000","last_unstale":"2024-04-25T19:44:11.043241+0000","last_undegraded":"2024-04-25T19:44:11.043241+0000","last_fullsized":"2024-04-25T19:44:11.043241+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:25.996441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043040+0000","last_change":"2024-04-25T19:44:10.050042+0000","last_active":"2024-04-25T19:44:11.043040+0000","last_peered":"2024-04-25T19:44:11.043040+0000","last_clean":"2024-04-25T19:44:11.043040+0000","last_became_active":"2024-04-25T19:44:10.049701+0000","last_became_peered":"2024-04-25T19:44:10.049701+0000","last_unstale":"2024-04-25T19:44:11.043040+0000","last_undegraded":"2024-04-25T19:44:11.043040+0000","last_fullsized":"2024-04-25T19:44:11.043040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:52:52.432869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043083+0000","last_change":"2024-04-25T19:44:09.040543+0000","last_active":"2024-04-25T19:44:11.043083+0000","last_peered":"2024-04-25T19:44:11.043083+0000","last_clean":"2024-04-25T19:44:11.043083+0000","last_became_active":"2024-04-25T19:44:09.040137+0000","last_became_peered":"2024-04-25T19:44:09.040137+0000","last_unstale":"2024-04-25T19:44:11.043083+0000","last_undegraded":"2024-04-25T19:44:11.043083+0000","last_fullsized":"2024-04-25T19:44:11.043083+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:59:23.033455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043278+0000","last_change":"2024-04-25T19:44:04.905598+0000","last_active":"2024-04-25T19:44:11.043278+0000","last_peered":"2024-04-25T19:44:11.043278+0000","last_clean":"2024-04-25T19:44:11.043278+0000","last_became_active":"2024-04-25T19:44:04.904778+0000","last_became_peered":"2024-04-25T19:44:04.904778+0000","last_unstale":"2024-04-25T19:44:11.043278+0000","last_undegraded":"2024-04-25T19:44:11.043278+0000","last_fullsized":"2024-04-25T19:44:11.043278+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:03:40.381610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745207+0000","last_change":"2024-04-25T19:44:10.063439+0000","last_active":"2024-04-25T19:44:11.745207+0000","last_peered":"2024-04-25T19:44:11.745207+0000","last_clean":"2024-04-25T19:44:11.745207+0000","last_became_active":"2024-04-25T19:44:10.062610+0000","last_became_peered":"2024-04-25T19:44:10.062610+0000","last_unstale":"2024-04-25T19:44:11.745207+0000","last_undegraded":"2024-04-25T19:44:11.745207+0000","last_fullsized":"2024-04-25T19:44:11.745207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:43:53.239881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043105+0000","last_change":"2024-04-25T19:44:09.044556+0000","last_active":"2024-04-25T19:44:11.043105+0000","last_peered":"2024-04-25T19:44:11.043105+0000","last_clean":"2024-04-25T19:44:11.043105+0000","last_became_active":"2024-04-25T19:44:09.044205+0000","last_became_peered":"2024-04-25T19:44:09.044205+0000","last_unstale":"2024-04-25T19:44:11.043105+0000","last_undegraded":"2024-04-25T19:44:11.043105+0000","last_fullsized":"2024-04-25T19:44:11.043105+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:59:23.033455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436963+0000","last_change":"2024-04-25T19:44:04.904842+0000","last_active":"2024-04-25T19:44:11.436963+0000","last_peered":"2024-04-25T19:44:11.436963+0000","last_clean":"2024-04-25T19:44:11.436963+0000","last_became_active":"2024-04-25T19:44:04.904449+0000","last_became_peered":"2024-04-25T19:44:04.904449+0000","last_unstale":"2024-04-25T19:44:11.436963+0000","last_undegraded":"2024-04-25T19:44:11.436963+0000","last_fullsized":"2024-04-25T19:44:11.436963+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:38.322685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.744965+0000","last_change":"2024-04-25T19:44:10.060164+0000","last_active":"2024-04-25T19:44:11.744965+0000","last_peered":"2024-04-25T19:44:11.744965+0000","last_clean":"2024-04-25T19:44:11.744965+0000","last_became_active":"2024-04-25T19:44:10.059272+0000","last_became_peered":"2024-04-25T19:44:10.059272+0000","last_unstale":"2024-04-25T19:44:11.744965+0000","last_undegraded":"2024-04-25T19:44:11.744965+0000","last_fullsized":"2024-04-25T19:44:11.744965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:49:57.885441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436380+0000","last_change":"2024-04-25T19:44:09.058552+0000","last_active":"2024-04-25T19:44:11.436380+0000","last_peered":"2024-04-25T19:44:11.436380+0000","last_clean":"2024-04-25T19:44:11.436380+0000","last_became_active":"2024-04-25T19:44:09.058107+0000","last_became_peered":"2024-04-25T19:44:09.058107+0000","last_unstale":"2024-04-25T19:44:11.436380+0000","last_undegraded":"2024-04-25T19:44:11.436380+0000","last_fullsized":"2024-04-25T19:44:11.436380+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:27.990801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042931+0000","last_change":"2024-04-25T19:44:04.904847+0000","last_active":"2024-04-25T19:44:11.042931+0000","last_peered":"2024-04-25T19:44:11.042931+0000","last_clean":"2024-04-25T19:44:11.042931+0000","last_became_active":"2024-04-25T19:44:04.904399+0000","last_became_peered":"2024-04-25T19:44:04.904399+0000","last_unstale":"2024-04-25T19:44:11.042931+0000","last_undegraded":"2024-04-25T19:44:11.042931+0000","last_fullsized":"2024-04-25T19:44:11.042931+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:31.155970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435636+0000","last_change":"2024-04-25T19:44:10.067351+0000","last_active":"2024-04-25T19:44:11.435636+0000","last_peered":"2024-04-25T19:44:11.435636+0000","last_clean":"2024-04-25T19:44:11.435636+0000","last_became_active":"2024-04-25T19:44:10.065833+0000","last_became_peered":"2024-04-25T19:44:10.065833+0000","last_unstale":"2024-04-25T19:44:11.435636+0000","last_undegraded":"2024-04-25T19:44:11.435636+0000","last_fullsized":"2024-04-25T19:44:11.435636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:41:44.189493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745703+0000","last_change":"2024-04-25T19:44:09.048708+0000","last_active":"2024-04-25T19:44:11.745703+0000","last_peered":"2024-04-25T19:44:11.745703+0000","last_clean":"2024-04-25T19:44:11.745703+0000","last_became_active":"2024-04-25T19:44:09.048320+0000","last_became_peered":"2024-04-25T19:44:09.048320+0000","last_unstale":"2024-04-25T19:44:11.745703+0000","last_undegraded":"2024-04-25T19:44:11.745703+0000","last_fullsized":"2024-04-25T19:44:11.745703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:48:47.178567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436571+0000","last_change":"2024-04-25T19:44:04.907462+0000","last_active":"2024-04-25T19:44:11.436571+0000","last_peered":"2024-04-25T19:44:11.436571+0000","last_clean":"2024-04-25T19:44:11.436571+0000","last_became_active":"2024-04-25T19:44:04.907134+0000","last_became_peered":"2024-04-25T19:44:04.907134+0000","last_unstale":"2024-04-25T19:44:11.436571+0000","last_undegraded":"2024-04-25T19:44:11.436571+0000","last_fullsized":"2024-04-25T19:44:11.436571+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:05:53.288205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042736+0000","last_change":"2024-04-25T19:44:10.062982+0000","last_active":"2024-04-25T19:44:11.042736+0000","last_peered":"2024-04-25T19:44:11.042736+0000","last_clean":"2024-04-25T19:44:11.042736+0000","last_became_active":"2024-04-25T19:44:10.062674+0000","last_became_peered":"2024-04-25T19:44:10.062674+0000","last_unstale":"2024-04-25T19:44:11.042736+0000","last_undegraded":"2024-04-25T19:44:11.042736+0000","last_fullsized":"2024-04-25T19:44:11.042736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:21.909648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043372+0000","last_change":"2024-04-25T19:44:10.056196+0000","last_active":"2024-04-25T19:44:11.043372+0000","last_peered":"2024-04-25T19:44:11.043372+0000","last_clean":"2024-04-25T19:44:11.043372+0000","last_became_active":"2024-04-25T19:44:10.055892+0000","last_became_peered":"2024-04-25T19:44:10.055892+0000","last_unstale":"2024-04-25T19:44:11.043372+0000","last_undegraded":"2024-04-25T19:44:11.043372+0000","last_fullsized":"2024-04-25T19:44:11.043372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:40.201502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436060+0000","last_change":"2024-04-25T19:44:09.056570+0000","last_active":"2024-04-25T19:44:11.436060+0000","last_peered":"2024-04-25T19:44:11.436060+0000","last_clean":"2024-04-25T19:44:11.436060+0000","last_became_active":"2024-04-25T19:44:09.055464+0000","last_became_peered":"2024-04-25T19:44:09.055464+0000","last_unstale":"2024-04-25T19:44:11.436060+0000","last_undegraded":"2024-04-25T19:44:11.436060+0000","last_fullsized":"2024-04-25T19:44:11.436060+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:48:00.496337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745172+0000","last_change":"2024-04-25T19:44:10.063199+0000","last_active":"2024-04-25T19:44:11.745172+0000","last_peered":"2024-04-25T19:44:11.745172+0000","last_clean":"2024-04-25T19:44:11.745172+0000","last_became_active":"2024-04-25T19:44:10.062127+0000","last_became_peered":"2024-04-25T19:44:10.062127+0000","last_unstale":"2024-04-25T19:44:11.745172+0000","last_undegraded":"2024-04-25T19:44:11.745172+0000","last_fullsized":"2024-04-25T19:44:11.745172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:36:33.677454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042922+0000","last_change":"2024-04-25T19:44:09.059522+0000","last_active":"2024-04-25T19:44:11.042922+0000","last_peered":"2024-04-25T19:44:11.042922+0000","last_clean":"2024-04-25T19:44:11.042922+0000","last_became_active":"2024-04-25T19:44:09.059182+0000","last_became_peered":"2024-04-25T19:44:09.059182+0000","last_unstale":"2024-04-25T19:44:11.042922+0000","last_undegraded":"2024-04-25T19:44:11.042922+0000","last_fullsized":"2024-04-25T19:44:11.042922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:25.699599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043346+0000","last_change":"2024-04-25T19:44:09.054779+0000","last_active":"2024-04-25T19:44:11.043346+0000","last_peered":"2024-04-25T19:44:11.043346+0000","last_clean":"2024-04-25T19:44:11.043346+0000","last_became_active":"2024-04-25T19:44:09.054451+0000","last_became_peered":"2024-04-25T19:44:09.054451+0000","last_unstale":"2024-04-25T19:44:11.043346+0000","last_undegraded":"2024-04-25T19:44:11.043346+0000","last_fullsized":"2024-04-25T19:44:11.043346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:59.352345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.041751+0000","last_change":"2024-04-25T19:44:10.043921+0000","last_active":"2024-04-25T19:44:11.041751+0000","last_peered":"2024-04-25T19:44:11.041751+0000","last_clean":"2024-04-25T19:44:11.041751+0000","last_became_active":"2024-04-25T19:44:10.043528+0000","last_became_peered":"2024-04-25T19:44:10.043528+0000","last_unstale":"2024-04-25T19:44:11.041751+0000","last_undegraded":"2024-04-25T19:44:11.041751+0000","last_fullsized":"2024-04-25T19:44:11.041751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:55:45.331783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745974+0000","last_change":"2024-04-25T19:44:09.055027+0000","last_active":"2024-04-25T19:44:11.745974+0000","last_peered":"2024-04-25T19:44:11.745974+0000","last_clean":"2024-04-25T19:44:11.745974+0000","last_became_active":"2024-04-25T19:44:09.054691+0000","last_became_peered":"2024-04-25T19:44:09.054691+0000","last_unstale":"2024-04-25T19:44:11.745974+0000","last_undegraded":"2024-04-25T19:44:11.745974+0000","last_fullsized":"2024-04-25T19:44:11.745974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:25.996441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435340+0000","last_change":"2024-04-25T19:44:10.067089+0000","last_active":"2024-04-25T19:44:11.435340+0000","last_peered":"2024-04-25T19:44:11.435340+0000","last_clean":"2024-04-25T19:44:11.435340+0000","last_became_active":"2024-04-25T19:44:10.065329+0000","last_became_peered":"2024-04-25T19:44:10.065329+0000","last_unstale":"2024-04-25T19:44:11.435340+0000","last_undegraded":"2024-04-25T19:44:11.435340+0000","last_fullsized":"2024-04-25T19:44:11.435340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:51.252629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436700+0000","last_change":"2024-04-25T19:44:10.060616+0000","last_active":"2024-04-25T19:44:11.436700+0000","last_peered":"2024-04-25T19:44:11.436700+0000","last_clean":"2024-04-25T19:44:11.436700+0000","last_became_active":"2024-04-25T19:44:10.060285+0000","last_became_peered":"2024-04-25T19:44:10.060285+0000","last_unstale":"2024-04-25T19:44:11.436700+0000","last_undegraded":"2024-04-25T19:44:11.436700+0000","last_fullsized":"2024-04-25T19:44:11.436700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:23:38.854656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435812+0000","last_change":"2024-04-25T19:44:09.059263+0000","last_active":"2024-04-25T19:44:11.435812+0000","last_peered":"2024-04-25T19:44:11.435812+0000","last_clean":"2024-04-25T19:44:11.435812+0000","last_became_active":"2024-04-25T19:44:09.058936+0000","last_became_peered":"2024-04-25T19:44:09.058936+0000","last_unstale":"2024-04-25T19:44:11.435812+0000","last_undegraded":"2024-04-25T19:44:11.435812+0000","last_fullsized":"2024-04-25T19:44:11.435812+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:49.562956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042882+0000","last_change":"2024-04-25T19:44:10.051152+0000","last_active":"2024-04-25T19:44:11.042882+0000","last_peered":"2024-04-25T19:44:11.042882+0000","last_clean":"2024-04-25T19:44:11.042882+0000","last_became_active":"2024-04-25T19:44:10.050607+0000","last_became_peered":"2024-04-25T19:44:10.050607+0000","last_unstale":"2024-04-25T19:44:11.042882+0000","last_undegraded":"2024-04-25T19:44:11.042882+0000","last_fullsized":"2024-04-25T19:44:11.042882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:28.992782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.744879+0000","last_change":"2024-04-25T19:44:09.054711+0000","last_active":"2024-04-25T19:44:11.744879+0000","last_peered":"2024-04-25T19:44:11.744879+0000","last_clean":"2024-04-25T19:44:11.744879+0000","last_became_active":"2024-04-25T19:44:09.053989+0000","last_became_peered":"2024-04-25T19:44:09.053989+0000","last_unstale":"2024-04-25T19:44:11.744879+0000","last_undegraded":"2024-04-25T19:44:11.744879+0000","last_fullsized":"2024-04-25T19:44:11.744879+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:25.699599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436322+0000","last_change":"2024-04-25T19:44:10.059956+0000","last_active":"2024-04-25T19:44:11.436322+0000","last_peered":"2024-04-25T19:44:11.436322+0000","last_clean":"2024-04-25T19:44:11.436322+0000","last_became_active":"2024-04-25T19:44:10.058700+0000","last_became_peered":"2024-04-25T19:44:10.058700+0000","last_unstale":"2024-04-25T19:44:11.436322+0000","last_undegraded":"2024-04-25T19:44:11.436322+0000","last_fullsized":"2024-04-25T19:44:11.436322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:25:09.001237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745309+0000","last_change":"2024-04-25T19:44:09.044537+0000","last_active":"2024-04-25T19:44:11.745309+0000","last_peered":"2024-04-25T19:44:11.745309+0000","last_clean":"2024-04-25T19:44:11.745309+0000","last_became_active":"2024-04-25T19:44:09.044205+0000","last_became_peered":"2024-04-25T19:44:09.044205+0000","last_unstale":"2024-04-25T19:44:11.745309+0000","last_undegraded":"2024-04-25T19:44:11.745309+0000","last_fullsized":"2024-04-25T19:44:11.745309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:11:22.034588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042844+0000","last_change":"2024-04-25T19:44:10.050365+0000","last_active":"2024-04-25T19:44:11.042844+0000","last_peered":"2024-04-25T19:44:11.042844+0000","last_clean":"2024-04-25T19:44:11.042844+0000","last_became_active":"2024-04-25T19:44:10.049730+0000","last_became_peered":"2024-04-25T19:44:10.049730+0000","last_unstale":"2024-04-25T19:44:11.042844+0000","last_undegraded":"2024-04-25T19:44:11.042844+0000","last_fullsized":"2024-04-25T19:44:11.042844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:26:17.991276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436186+0000","last_change":"2024-04-25T19:44:09.058547+0000","last_active":"2024-04-25T19:44:11.436186+0000","last_peered":"2024-04-25T19:44:11.436186+0000","last_clean":"2024-04-25T19:44:11.436186+0000","last_became_active":"2024-04-25T19:44:09.057348+0000","last_became_peered":"2024-04-25T19:44:09.057348+0000","last_unstale":"2024-04-25T19:44:11.436186+0000","last_undegraded":"2024-04-25T19:44:11.436186+0000","last_fullsized":"2024-04-25T19:44:11.436186+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:38:52.992542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042693+0000","last_change":"2024-04-25T19:44:10.047756+0000","last_active":"2024-04-25T19:44:11.042693+0000","last_peered":"2024-04-25T19:44:11.042693+0000","last_clean":"2024-04-25T19:44:11.042693+0000","last_became_active":"2024-04-25T19:44:10.047155+0000","last_became_peered":"2024-04-25T19:44:10.047155+0000","last_unstale":"2024-04-25T19:44:11.042693+0000","last_undegraded":"2024-04-25T19:44:11.042693+0000","last_fullsized":"2024-04-25T19:44:11.042693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:43:23.290058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745329+0000","last_change":"2024-04-25T19:44:09.053455+0000","last_active":"2024-04-25T19:44:11.745329+0000","last_peered":"2024-04-25T19:44:11.745329+0000","last_clean":"2024-04-25T19:44:11.745329+0000","last_became_active":"2024-04-25T19:44:09.053118+0000","last_became_peered":"2024-04-25T19:44:09.053118+0000","last_unstale":"2024-04-25T19:44:11.745329+0000","last_undegraded":"2024-04-25T19:44:11.745329+0000","last_fullsized":"2024-04-25T19:44:11.745329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:59:23.033455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043396+0000","last_change":"2024-04-25T19:44:10.047937+0000","last_active":"2024-04-25T19:44:11.043396+0000","last_peered":"2024-04-25T19:44:11.043396+0000","last_clean":"2024-04-25T19:44:11.043396+0000","last_became_active":"2024-04-25T19:44:10.046231+0000","last_became_peered":"2024-04-25T19:44:10.046231+0000","last_unstale":"2024-04-25T19:44:11.043396+0000","last_undegraded":"2024-04-25T19:44:11.043396+0000","last_fullsized":"2024-04-25T19:44:11.043396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:50.564937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436779+0000","last_change":"2024-04-25T19:44:09.060536+0000","last_active":"2024-04-25T19:44:11.436779+0000","last_peered":"2024-04-25T19:44:11.436779+0000","last_clean":"2024-04-25T19:44:11.436779+0000","last_became_active":"2024-04-25T19:44:09.059473+0000","last_became_peered":"2024-04-25T19:44:09.059473+0000","last_unstale":"2024-04-25T19:44:11.436779+0000","last_undegraded":"2024-04-25T19:44:11.436779+0000","last_fullsized":"2024-04-25T19:44:11.436779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:20.907667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435968+0000","last_change":"2024-04-25T19:44:10.043795+0000","last_active":"2024-04-25T19:44:11.435968+0000","last_peered":"2024-04-25T19:44:11.435968+0000","last_clean":"2024-04-25T19:44:11.435968+0000","last_became_active":"2024-04-25T19:44:10.043342+0000","last_became_peered":"2024-04-25T19:44:10.043342+0000","last_unstale":"2024-04-25T19:44:11.435968+0000","last_undegraded":"2024-04-25T19:44:11.435968+0000","last_fullsized":"2024-04-25T19:44:11.435968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:52:49.026097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042861+0000","last_change":"2024-04-25T19:44:09.044583+0000","last_active":"2024-04-25T19:44:11.042861+0000","last_peered":"2024-04-25T19:44:11.042861+0000","last_clean":"2024-04-25T19:44:11.042861+0000","last_became_active":"2024-04-25T19:44:09.044225+0000","last_became_peered":"2024-04-25T19:44:09.044225+0000","last_unstale":"2024-04-25T19:44:11.042861+0000","last_undegraded":"2024-04-25T19:44:11.042861+0000","last_fullsized":"2024-04-25T19:44:11.042861+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:25.699599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042456+0000","last_change":"2024-04-25T19:44:10.049048+0000","last_active":"2024-04-25T19:44:11.042456+0000","last_peered":"2024-04-25T19:44:11.042456+0000","last_clean":"2024-04-25T19:44:11.042456+0000","last_became_active":"2024-04-25T19:44:10.048740+0000","last_became_peered":"2024-04-25T19:44:10.048740+0000","last_unstale":"2024-04-25T19:44:11.042456+0000","last_undegraded":"2024-04-25T19:44:11.042456+0000","last_fullsized":"2024-04-25T19:44:11.042456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:19.654106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745972+0000","last_change":"2024-04-25T19:44:09.046229+0000","last_active":"2024-04-25T19:44:11.745972+0000","last_peered":"2024-04-25T19:44:11.745972+0000","last_clean":"2024-04-25T19:44:11.745972+0000","last_became_active":"2024-04-25T19:44:09.045423+0000","last_became_peered":"2024-04-25T19:44:09.045423+0000","last_unstale":"2024-04-25T19:44:11.745972+0000","last_undegraded":"2024-04-25T19:44:11.745972+0000","last_fullsized":"2024-04-25T19:44:11.745972+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:30:43.807694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672966,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":64550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":654870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672966,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":790528,"data_stored":653786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":188416,"data_stored":63326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T19:44:14.464 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T19:44:14.464 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T19:44:14.464 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-25T19:44:14.465 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-25T19:44:14.465 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-25T19:44:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.610+0000 7f4fe203a640 1 Processor -- start 2024-04-25T19:44:14.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.610+0000 7f4fe203a640 1 -- start start 2024-04-25T19:44:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.611+0000 7f4fe203a640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4fdc0ad450 0x7f4fdc152400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.611+0000 7f4fe203a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fdc146c60 con 0x7f4fdc06da90 2024-04-25T19:44:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.611+0000 7f4fe203a640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fdc0ad9d0 con 0x7f4fdc0a56d0 2024-04-25T19:44:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.611+0000 7f4fe203a640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f4fdc1479e0 con 0x7f4fdc0ad450 2024-04-25T19:44:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.611+0000 7f4fdb7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4fdc06da90 0x7f4fdc1472d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37998/0 (socket says 172.21.15.29:37998) 2024-04-25T19:44:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.611+0000 7f4fdb7fe640 1 -- 172.21.15.29:0/2586286972 learned_addr learned my addr 172.21.15.29:0/2586286972 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.612+0000 7f4fdbfff640 1 --2- 172.21.15.29:0/2586286972 >> v2:172.21.15.102:3300/0 conn(0x7f4fdc0ad450 0x7f4fdc152400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.612+0000 7f4fda7fc640 1 -- 172.21.15.29:0/2586286972 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1549339587 0 0) 0x7f4fdc146c60 con 0x7f4fdc06da90 2024-04-25T19:44:14.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.612+0000 7f4fda7fc640 1 -- 172.21.15.29:0/2586286972 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4fc4003520 con 0x7f4fdc06da90 2024-04-25T19:44:14.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.612+0000 7f4fdbfff640 1 -- 172.21.15.29:0/2586286972 >> v1:172.21.15.102:6789/0 conn(0x7f4fdc0a56d0 legacy=0x7f4fdc0acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:14.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.612+0000 7f4fda7fc640 1 -- 172.21.15.29:0/2586286972 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 722178699 0 0) 0x7f4fdc0ad9d0 con 0x7f4fdc0a56d0 2024-04-25T19:44:14.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.612+0000 7f4fdbfff640 1 -- 172.21.15.29:0/2586286972 >> v1:172.21.15.29:6789/0 conn(0x7f4fdc06da90 legacy=0x7f4fdc1472d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:14.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.612+0000 7f4fdbfff640 1 -- 172.21.15.29:0/2586286972 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fdc0a5ae0 con 0x7f4fdc0ad450 2024-04-25T19:44:14.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.613+0000 7f4fdbfff640 1 --2- 172.21.15.29:0/2586286972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fdc0ad450 0x7f4fdc152400 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4009d90 tx=0x7f4fd402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11ec6c7429d70991 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:14.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.613+0000 7f4fda7fc640 1 -- 172.21.15.29:0/2586286972 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd4039070 con 0x7f4fdc0ad450 2024-04-25T19:44:14.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.613+0000 7f4fda7fc640 1 -- 172.21.15.29:0/2586286972 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fd402cd00 con 0x7f4fdc0ad450 2024-04-25T19:44:14.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.613+0000 7f4fda7fc640 1 -- 172.21.15.29:0/2586286972 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd4034660 con 0x7f4fdc0ad450 2024-04-25T19:44:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.614+0000 7f4fe203a640 1 -- 172.21.15.29:0/2586286972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fdc0ad450 msgr2=0x7f4fdc152400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.614+0000 7f4fe203a640 1 --2- 172.21.15.29:0/2586286972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fdc0ad450 0x7f4fdc152400 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4009d90 tx=0x7f4fd402bf50 comp rx=0 tx=0).stop 2024-04-25T19:44:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.614+0000 7f4fe203a640 1 -- 172.21.15.29:0/2586286972 shutdown_connections 2024-04-25T19:44:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.614+0000 7f4fe203a640 1 --2- 172.21.15.29:0/2586286972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fdc0ad450 0x7f4fdc152400 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.614+0000 7f4fe203a640 1 -- 172.21.15.29:0/2586286972 >> 172.21.15.29:0/2586286972 conn(0x7f4fdc0bc3b0 msgr2=0x7f4fdc0bc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.614+0000 7f4fe203a640 1 -- 172.21.15.29:0/2586286972 shutdown_connections 2024-04-25T19:44:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.614+0000 7f4fe203a640 1 -- 172.21.15.29:0/2586286972 wait complete. 2024-04-25T19:44:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.615+0000 7f4fe203a640 1 Processor -- start 2024-04-25T19:44:14.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.615+0000 7f4fe203a640 1 -- start start 2024-04-25T19:44:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.616+0000 7f4fe203a640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fdc06da90 0x7f4fdc0a1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.616+0000 7f4fe203a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fdc0a56d0 0x7f4fdc09cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.616+0000 7f4fe203a640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4fdc0ad450 0x7f4fdc09d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.616+0000 7f4fe203a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4fdc152cd0 con 0x7f4fdc0a56d0 2024-04-25T19:44:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.616+0000 7f4fe203a640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4fdc152b50 con 0x7f4fdc06da90 2024-04-25T19:44:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.616+0000 7f4fe203a640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f4fdc152e50 con 0x7f4fdc0ad450 2024-04-25T19:44:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.616+0000 7f4fdaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fdc0a56d0 0x7f4fdc09cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.617+0000 7f4fdaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fdc0a56d0 0x7f4fdc09cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36516/0 (socket says 172.21.15.29:36516) 2024-04-25T19:44:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.617+0000 7f4fdbfff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4fdc0ad450 0x7f4fdc09d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.617+0000 7f4fdaffd640 1 -- 172.21.15.29:0/452740370 learned_addr learned my addr 172.21.15.29:0/452740370 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.617+0000 7f4fdb7fe640 1 --2- 172.21.15.29:0/452740370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fdc06da90 0x7f4fdc0a1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.617+0000 7f4fdbfff640 1 -- 172.21.15.29:0/452740370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fdc06da90 msgr2=0x7f4fdc0a1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.617+0000 7f4fdbfff640 1 --2- 172.21.15.29:0/452740370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fdc06da90 0x7f4fdc0a1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.617+0000 7f4fdbfff640 1 -- 172.21.15.29:0/452740370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fdc0a56d0 msgr2=0x7f4fdc09cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.617+0000 7f4fdbfff640 1 --2- 172.21.15.29:0/452740370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fdc0a56d0 0x7f4fdc09cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.617+0000 7f4fdbfff640 1 -- 172.21.15.29:0/452740370 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fdc09e840 con 0x7f4fdc0ad450 2024-04-25T19:44:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.617+0000 7f4fdaffd640 1 --2- 172.21.15.29:0/452740370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fdc0a56d0 0x7f4fdc09cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.617+0000 7f4fdbfff640 1 --2- 172.21.15.29:0/452740370 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4fdc0ad450 0x7f4fdc09d330 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4033000 tx=0x7f4fd4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.617+0000 7f4fc3fff640 1 -- 172.21.15.29:0/452740370 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd4039050 con 0x7f4fdc0ad450 2024-04-25T19:44:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.617+0000 7f4fc3fff640 1 -- 172.21.15.29:0/452740370 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fd403a040 con 0x7f4fdc0ad450 2024-04-25T19:44:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.617+0000 7f4fe203a640 1 -- 172.21.15.29:0/452740370 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fdc09f3d0 con 0x7f4fdc0ad450 2024-04-25T19:44:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.617+0000 7f4fc3fff640 1 -- 172.21.15.29:0/452740370 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd4009450 con 0x7f4fdc0ad450 2024-04-25T19:44:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.617+0000 7f4fe203a640 1 -- 172.21.15.29:0/452740370 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fdc0a2010 con 0x7f4fdc0ad450 2024-04-25T19:44:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.618+0000 7f4fe203a640 1 -- 172.21.15.29:0/452740370 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fdc09f7d0 con 0x7f4fdc0ad450 2024-04-25T19:44:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.620+0000 7f4fc3fff640 1 -- 172.21.15.29:0/452740370 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f4fd4047020 con 0x7f4fdc0ad450 2024-04-25T19:44:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.620+0000 7f4fc3fff640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.620+0000 7f4fc3fff640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.620+0000 7f4fc3fff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.620+0000 7f4fc3fff640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.620+0000 7f4fc3fff640 1 --2- 172.21.15.29:0/452740370 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f4fc80ac530 0x7f4fc80aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.620+0000 7f4fc3fff640 1 -- 172.21.15.29:0/452740370 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f4fd40fc180 con 0x7f4fdc0ad450 2024-04-25T19:44:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.620+0000 7f4fdb7fe640 1 --2- 172.21.15.29:0/452740370 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f4fc80ac530 0x7f4fc80aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.621+0000 7f4fdb7fe640 1 --2- 172.21.15.29:0/452740370 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f4fc80ac530 0x7f4fc80aea40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc005880 tx=0x7f4fcc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:14.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.623+0000 7f4fc3fff640 1 -- 172.21.15.29:0/452740370 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4fd40c6df0 con 0x7f4fdc0ad450 2024-04-25T19:44:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.857+0000 7f4fe203a640 1 -- 172.21.15.29:0/452740370 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f4fdc0a8800 con 0x7f4fdc0ad450 2024-04-25T19:44:14.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.863+0000 7f4fc3fff640 1 -- 172.21.15.29:0/452740370 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+279392 (secure 0 0 0) 0x7f4fd40cde20 con 0x7f4fdc0ad450 2024-04-25T19:44:14.867 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:14.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.866+0000 7f4fe203a640 10 mgrc shutdown 2024-04-25T19:44:14.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.866+0000 7f4fe203a640 1 -- 172.21.15.29:0/452740370 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f4fc80ac530 msgr2=0x7f4fc80aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:14.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.866+0000 7f4fe203a640 1 --2- 172.21.15.29:0/452740370 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f4fc80ac530 0x7f4fc80aea40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc005880 tx=0x7f4fcc009040 comp rx=0 tx=0).stop 2024-04-25T19:44:14.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.866+0000 7f4fe203a640 1 -- 172.21.15.29:0/452740370 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4fdc0ad450 msgr2=0x7f4fdc09d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:14.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.866+0000 7f4fe203a640 1 --2- 172.21.15.29:0/452740370 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4fdc0ad450 0x7f4fdc09d330 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4033000 tx=0x7f4fd4004a40 comp rx=0 tx=0).stop 2024-04-25T19:44:14.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.866+0000 7f4fe203a640 1 -- 172.21.15.29:0/452740370 shutdown_connections 2024-04-25T19:44:14.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.866+0000 7f4fe203a640 1 --2- 172.21.15.29:0/452740370 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f4fc80ac530 0x7f4fc80aea40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.866+0000 7f4fe203a640 1 --2- 172.21.15.29:0/452740370 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4fdc0ad450 0x7f4fdc09d330 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.866+0000 7f4fe203a640 1 --2- 172.21.15.29:0/452740370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fdc0a56d0 0x7f4fdc09cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.866+0000 7f4fe203a640 1 --2- 172.21.15.29:0/452740370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fdc06da90 0x7f4fdc0a1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:14.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.866+0000 7f4fe203a640 1 -- 172.21.15.29:0/452740370 >> 172.21.15.29:0/452740370 conn(0x7f4fdc0bc3b0 msgr2=0x7f4fdc1460c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:14.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.866+0000 7f4fe203a640 1 -- 172.21.15.29:0/452740370 shutdown_connections 2024-04-25T19:44:14.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:14.866+0000 7f4fe203a640 1 -- 172.21.15.29:0/452740370 wait complete. 2024-04-25T19:44:14.878 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":5,"flags":0,"active_gid":4108,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1439704316},{"type":"v1","addr":"172.21.15.102:6825","nonce":1439704316}]},"active_addr":"172.21.15.102:6825/1439704316","active_change":"2024-04-25T19:43:52.073940+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4100,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":4111,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":2274001855}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":1318055012}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":4278965720}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":1813070992}]}]} 2024-04-25T19:44:14.885 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-25T19:44:14.885 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-25T19:44:14.885 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:44:15.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.027+0000 7f1ed11db640 1 Processor -- start 2024-04-25T19:44:15.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.027+0000 7f1ed11db640 1 -- start start 2024-04-25T19:44:15.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.028+0000 7f1ed11db640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ecc0ba100 0x7f1ecc0addb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.028+0000 7f1ed11db640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ecc137650 0x7f1ecc0b7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.028+0000 7f1ed11db640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ecc0ae2f0 con 0x7f1ecc0b70f0 2024-04-25T19:44:15.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.028+0000 7f1ed11db640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f1ecc137e90 con 0x7f1ecc137650 2024-04-25T19:44:15.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.028+0000 7f1ed11db640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f1ecc0ba510 con 0x7f1ecc0ba100 2024-04-25T19:44:15.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.029+0000 7f1ecad9d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ecc0ba100 0x7f1ecc0addb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.029+0000 7f1ecb59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ecc137650 0x7f1ecc0b7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.029+0000 7f1eca59c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1ecc0b70f0 0x7f1ecc136f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38002/0 (socket says 172.21.15.29:38002) 2024-04-25T19:44:15.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.029+0000 7f1eca59c640 1 -- 172.21.15.29:0/1006609335 learned_addr learned my addr 172.21.15.29:0/1006609335 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:15.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.029+0000 7f1ecb59e640 1 -- 172.21.15.29:0/1006609335 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ecc0ba100 msgr2=0x7f1ecc0addb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:15.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.029+0000 7f1ecb59e640 1 --2- 172.21.15.29:0/1006609335 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ecc0ba100 0x7f1ecc0addb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.029+0000 7f1ecb59e640 1 -- 172.21.15.29:0/1006609335 >> v1:172.21.15.29:6789/0 conn(0x7f1ecc0b70f0 legacy=0x7f1ecc136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.029+0000 7f1ecb59e640 1 -- 172.21.15.29:0/1006609335 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ecc137bd0 con 0x7f1ecc137650 2024-04-25T19:44:15.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.029+0000 7f1ec9d9b640 1 -- 172.21.15.29:0/1006609335 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3693321501 0 0) 0x7f1ecc0ae2f0 con 0x7f1ecc0b70f0 2024-04-25T19:44:15.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.029+0000 7f1ecad9d640 1 --2- 172.21.15.29:0/1006609335 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ecc0ba100 0x7f1ecc0addb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:15.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.030+0000 7f1ecb59e640 1 --2- 172.21.15.29:0/1006609335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ecc137650 0x7f1ecc0b7500 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc009d90 tx=0x7f1ebc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3615b7f1bf68bcbb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.030+0000 7f1ec9d9b640 1 -- 172.21.15.29:0/1006609335 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc039070 con 0x7f1ecc137650 2024-04-25T19:44:15.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.030+0000 7f1ec9d9b640 1 -- 172.21.15.29:0/1006609335 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ebc02cc80 con 0x7f1ecc137650 2024-04-25T19:44:15.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.030+0000 7f1ec9d9b640 1 -- 172.21.15.29:0/1006609335 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc034660 con 0x7f1ecc137650 2024-04-25T19:44:15.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.030+0000 7f1ed11db640 1 -- 172.21.15.29:0/1006609335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ecc137650 msgr2=0x7f1ecc0b7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.030+0000 7f1ed11db640 1 --2- 172.21.15.29:0/1006609335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ecc137650 0x7f1ecc0b7500 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc009d90 tx=0x7f1ebc02bed0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.031+0000 7f1ed11db640 1 -- 172.21.15.29:0/1006609335 shutdown_connections 2024-04-25T19:44:15.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.031+0000 7f1ed11db640 1 --2- 172.21.15.29:0/1006609335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ecc137650 0x7f1ecc0b7500 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.031+0000 7f1ed11db640 1 --2- 172.21.15.29:0/1006609335 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ecc0ba100 0x7f1ecc0addb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.031+0000 7f1ed11db640 1 -- 172.21.15.29:0/1006609335 >> 172.21.15.29:0/1006609335 conn(0x7f1ecc072b90 msgr2=0x7f1ecc0a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:15.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.031+0000 7f1ed11db640 1 -- 172.21.15.29:0/1006609335 shutdown_connections 2024-04-25T19:44:15.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.031+0000 7f1ed11db640 1 -- 172.21.15.29:0/1006609335 wait complete. 2024-04-25T19:44:15.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.032+0000 7f1ed11db640 1 Processor -- start 2024-04-25T19:44:15.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.032+0000 7f1ed11db640 1 -- start start 2024-04-25T19:44:15.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.033+0000 7f1ed11db640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ecc0b70f0 0x7f1ecc12e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.033+0000 7f1ed11db640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ecc0ba100 0x7f1ecc12d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.033+0000 7f1ed11db640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ecc137650 0x7f1ecc1387a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.033+0000 7f1ed11db640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1ecc14f630 con 0x7f1ecc0b70f0 2024-04-25T19:44:15.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.033+0000 7f1ed11db640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1ecc14f4b0 con 0x7f1ecc137650 2024-04-25T19:44:15.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.033+0000 7f1ed11db640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f1ecc14f7b0 con 0x7f1ecc0ba100 2024-04-25T19:44:15.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.033+0000 7f1ecad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ecc0b70f0 0x7f1ecc12e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.033+0000 7f1ecad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ecc0b70f0 0x7f1ecc12e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36528/0 (socket says 172.21.15.29:36528) 2024-04-25T19:44:15.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.033+0000 7f1eca59c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ecc0ba100 0x7f1ecc12d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.033+0000 7f1ecad9d640 1 -- 172.21.15.29:0/3453080447 learned_addr learned my addr 172.21.15.29:0/3453080447 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:15.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.034+0000 7f1ecb59e640 1 --2- 172.21.15.29:0/3453080447 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ecc137650 0x7f1ecc1387a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.034+0000 7f1ecad9d640 1 -- 172.21.15.29:0/3453080447 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ecc0ba100 msgr2=0x7f1ecc12d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.034+0000 7f1ecad9d640 1 --2- 172.21.15.29:0/3453080447 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ecc0ba100 0x7f1ecc12d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.034+0000 7f1ecad9d640 1 -- 172.21.15.29:0/3453080447 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ecc137650 msgr2=0x7f1ecc1387a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.034+0000 7f1ecad9d640 1 --2- 172.21.15.29:0/3453080447 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ecc137650 0x7f1ecc1387a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.034+0000 7f1ecad9d640 1 -- 172.21.15.29:0/3453080447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ecc133690 con 0x7f1ecc0b70f0 2024-04-25T19:44:15.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.034+0000 7f1eca59c640 1 --2- 172.21.15.29:0/3453080447 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ecc0ba100 0x7f1ecc12d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:15.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.034+0000 7f1ecad9d640 1 --2- 172.21.15.29:0/3453080447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ecc0b70f0 0x7f1ecc12e5a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1ec400e940 tx=0x7f1ec400ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.034+0000 7f1ec37fe640 1 -- 172.21.15.29:0/3453080447 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec4017070 con 0x7f1ecc0b70f0 2024-04-25T19:44:15.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.034+0000 7f1ec37fe640 1 -- 172.21.15.29:0/3453080447 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ec400ca00 con 0x7f1ecc0b70f0 2024-04-25T19:44:15.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.034+0000 7f1ed11db640 1 -- 172.21.15.29:0/3453080447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ecc12d680 con 0x7f1ecc0b70f0 2024-04-25T19:44:15.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.034+0000 7f1ec37fe640 1 -- 172.21.15.29:0/3453080447 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec400cdd0 con 0x7f1ecc0b70f0 2024-04-25T19:44:15.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.034+0000 7f1ed11db640 1 -- 172.21.15.29:0/3453080447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ecc12ce40 con 0x7f1ecc0b70f0 2024-04-25T19:44:15.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.035+0000 7f1ed11db640 1 -- 172.21.15.29:0/3453080447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ecc0507b0 con 0x7f1ecc0b70f0 2024-04-25T19:44:15.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.037+0000 7f1ec37fe640 1 -- 172.21.15.29:0/3453080447 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f1ec401a070 con 0x7f1ecc0b70f0 2024-04-25T19:44:15.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.037+0000 7f1ec37fe640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:15.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.037+0000 7f1ec37fe640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:15.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.037+0000 7f1ec37fe640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:15.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.037+0000 7f1ec37fe640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:15.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.037+0000 7f1ec37fe640 1 --2- 172.21.15.29:0/3453080447 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f1eac0ac560 0x7f1eac0aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.038+0000 7f1ec37fe640 1 -- 172.21.15.29:0/3453080447 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f1ec40d9990 con 0x7f1ecc0b70f0 2024-04-25T19:44:15.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.038+0000 7f1eca59c640 1 --2- 172.21.15.29:0/3453080447 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f1eac0ac560 0x7f1eac0aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.038+0000 7f1eca59c640 1 --2- 172.21.15.29:0/3453080447 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f1eac0ac560 0x7f1eac0aea70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4007fd0 tx=0x7f1eb4007de0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.041+0000 7f1ec37fe640 1 -- 172.21.15.29:0/3453080447 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1ec40a4600 con 0x7f1ecc0b70f0 2024-04-25T19:44:15.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.232+0000 7f1ed11db640 1 -- 172.21.15.29:0/3453080447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1ecc0b94b0 con 0x7f1ecc0b70f0 2024-04-25T19:44:15.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.233+0000 7f1ec37fe640 1 -- 172.21.15.29:0/3453080447 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+13512 (secure 0 0 0) 0x7f1ec40a3f00 con 0x7f1ecc0b70f0 2024-04-25T19:44:15.234 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:15.234 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"1b3c39b5-e787-409e-b0bd-f4f491c89d72","created":"2024-04-25T19:43:47.854697+0000","modified":"2024-04-25T19:44:11.035019+0000","last_up_change":"2024-04-25T19:43:57.853237+0000","last_in_change":"2024-04-25T19:43:49.022776+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:43:58.096700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T19:44:03.617011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:44:07.890493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2024-04-25T19:44:08.355570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d805223-ee84-494f-bd31-59091906aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6801","nonce":695704147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6803","nonce":695704147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6807","nonce":695704147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6805","nonce":695704147}]},"public_addr":"172.21.15.29:6801/695704147","cluster_addr":"172.21.15.29:6803/695704147","heartbeat_back_addr":"172.21.15.29:6807/695704147","heartbeat_front_addr":"172.21.15.29:6805/695704147","state":["exists","up"]},{"osd":1,"uuid":"607c5038-38af-440f-8371-e71758549920","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6814","nonce":3135821452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6817","nonce":3135821452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6823","nonce":3135821452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6821","nonce":3135821452}]},"public_addr":"172.21.15.102:6814/3135821452","cluster_addr":"172.21.15.102:6817/3135821452","heartbeat_back_addr":"172.21.15.102:6823/3135821452","heartbeat_front_addr":"172.21.15.102:6821/3135821452","state":["exists","up"]},{"osd":2,"uuid":"6859ae76-54bb-4c55-b0dc-5e2735c0d31c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6801","nonce":2071844144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6803","nonce":2071844144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6811","nonce":2071844144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6806","nonce":2071844144}]},"public_addr":"172.21.15.102:6801/2071844144","cluster_addr":"172.21.15.102:6803/2071844144","heartbeat_back_addr":"172.21.15.102:6811/2071844144","heartbeat_front_addr":"172.21.15.102:6806/2071844144","state":["exists","up"]},{"osd":3,"uuid":"a98b7b88-5206-4f22-84a6-78212e0d5f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6805","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6807","nonce":339944984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6810","nonce":339944984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6820","nonce":339944984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6815","nonce":339944984}]},"public_addr":"172.21.15.102:6807/339944984","cluster_addr":"172.21.15.102:6810/339944984","heartbeat_back_addr":"172.21.15.102:6820/339944984","heartbeat_front_addr":"172.21.15.102:6815/339944984","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.121529+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.201155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.264387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.198713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:44:15.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.235+0000 7f1ed11db640 10 mgrc shutdown 2024-04-25T19:44:15.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.235+0000 7f1ed11db640 1 -- 172.21.15.29:0/3453080447 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f1eac0ac560 msgr2=0x7f1eac0aea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.235+0000 7f1ed11db640 1 --2- 172.21.15.29:0/3453080447 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f1eac0ac560 0x7f1eac0aea70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4007fd0 tx=0x7f1eb4007de0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.235+0000 7f1ed11db640 1 -- 172.21.15.29:0/3453080447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ecc0b70f0 msgr2=0x7f1ecc12e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.235+0000 7f1ed11db640 1 --2- 172.21.15.29:0/3453080447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ecc0b70f0 0x7f1ecc12e5a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1ec400e940 tx=0x7f1ec400ee00 comp rx=0 tx=0).stop 2024-04-25T19:44:15.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.235+0000 7f1ed11db640 1 -- 172.21.15.29:0/3453080447 shutdown_connections 2024-04-25T19:44:15.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.235+0000 7f1ed11db640 1 --2- 172.21.15.29:0/3453080447 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f1eac0ac560 0x7f1eac0aea70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.235+0000 7f1ed11db640 1 --2- 172.21.15.29:0/3453080447 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ecc137650 0x7f1ecc1387a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.235+0000 7f1ed11db640 1 --2- 172.21.15.29:0/3453080447 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ecc0ba100 0x7f1ecc12d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.235+0000 7f1ed11db640 1 --2- 172.21.15.29:0/3453080447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ecc0b70f0 0x7f1ecc12e5a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.235+0000 7f1ed11db640 1 -- 172.21.15.29:0/3453080447 >> 172.21.15.29:0/3453080447 conn(0x7f1ecc072b90 msgr2=0x7f1ecc0ba6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:15.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.235+0000 7f1ed11db640 1 -- 172.21.15.29:0/3453080447 shutdown_connections 2024-04-25T19:44:15.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.235+0000 7f1ed11db640 1 -- 172.21.15.29:0/3453080447 wait complete. 2024-04-25T19:44:15.248 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-25T19:44:15.248 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:44:15.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.357+0000 7fbba5730640 1 Processor -- start 2024-04-25T19:44:15.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.357+0000 7fbba5730640 1 -- start start 2024-04-25T19:44:15.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.358+0000 7fbba5730640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbba006df40 0x7fbba00aca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.358+0000 7fbba5730640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbba00ad500 con 0x7fbba00acf70 2024-04-25T19:44:15.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.358+0000 7fbba5730640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbba00ae650 con 0x7fbba0146640 2024-04-25T19:44:15.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.358+0000 7fbba5730640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fbba00ad7c0 con 0x7fbba006df40 2024-04-25T19:44:15.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.358+0000 7fbb9f7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbba00acf70 0x7fbba0151400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38008/0 (socket says 172.21.15.29:38008) 2024-04-25T19:44:15.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.358+0000 7fbb9f7fe640 1 -- 172.21.15.29:0/876798566 learned_addr learned my addr 172.21.15.29:0/876798566 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:15.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.358+0000 7fbb9e7fc640 1 --2- 172.21.15.29:0/876798566 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbba006df40 0x7fbba00aca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.358+0000 7fbb9dffb640 1 -- 172.21.15.29:0/876798566 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1124404802 0 0) 0x7fbba00ad500 con 0x7fbba00acf70 2024-04-25T19:44:15.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.358+0000 7fbb9e7fc640 1 -- 172.21.15.29:0/876798566 >> v1:172.21.15.102:6789/0 conn(0x7fbba0146640 legacy=0x7fbba00aa840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.358+0000 7fbb9e7fc640 1 -- 172.21.15.29:0/876798566 >> v1:172.21.15.29:6789/0 conn(0x7fbba00acf70 legacy=0x7fbba0151400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.358+0000 7fbb9e7fc640 1 -- 172.21.15.29:0/876798566 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba006e350 con 0x7fbba006df40 2024-04-25T19:44:15.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.359+0000 7fbb9e7fc640 1 --2- 172.21.15.29:0/876798566 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbba006df40 0x7fbba00aca30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbb940097d0 tx=0x7fbb9402b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bff868501b322ae0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.359+0000 7fbb9dffb640 1 -- 172.21.15.29:0/876798566 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb94034070 con 0x7fbba006df40 2024-04-25T19:44:15.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.359+0000 7fbb9dffb640 1 -- 172.21.15.29:0/876798566 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb94030880 con 0x7fbba006df40 2024-04-25T19:44:15.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.359+0000 7fbb9dffb640 1 -- 172.21.15.29:0/876798566 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb94030c50 con 0x7fbba006df40 2024-04-25T19:44:15.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.360+0000 7fbba5730640 1 -- 172.21.15.29:0/876798566 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbba006df40 msgr2=0x7fbba00aca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.360+0000 7fbba5730640 1 --2- 172.21.15.29:0/876798566 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbba006df40 0x7fbba00aca30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbb940097d0 tx=0x7fbb9402b9c0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.360+0000 7fbba5730640 1 -- 172.21.15.29:0/876798566 shutdown_connections 2024-04-25T19:44:15.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.360+0000 7fbba5730640 1 --2- 172.21.15.29:0/876798566 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbba006df40 0x7fbba00aca30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.360+0000 7fbba5730640 1 -- 172.21.15.29:0/876798566 >> 172.21.15.29:0/876798566 conn(0x7fbba00ab710 msgr2=0x7fbba00bbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:15.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.360+0000 7fbba5730640 1 -- 172.21.15.29:0/876798566 shutdown_connections 2024-04-25T19:44:15.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.360+0000 7fbba5730640 1 -- 172.21.15.29:0/876798566 wait complete. 2024-04-25T19:44:15.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.362+0000 7fbba5730640 1 Processor -- start 2024-04-25T19:44:15.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.362+0000 7fbba5730640 1 -- start start 2024-04-25T19:44:15.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbba5730640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbba006df40 0x7fbba0148a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbba5730640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbba00acf70 0x7fbba013a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbba5730640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbba0146640 0x7fbba01e0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbba5730640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbba0152eb0 con 0x7fbba00acf70 2024-04-25T19:44:15.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbba5730640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbba0152d30 con 0x7fbba006df40 2024-04-25T19:44:15.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbba5730640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fbba0153030 con 0x7fbba0146640 2024-04-25T19:44:15.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbb9e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbba00acf70 0x7fbba013a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbb9f7fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbba0146640 0x7fbba01e0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbb9e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbba00acf70 0x7fbba013a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36534/0 (socket says 172.21.15.29:36534) 2024-04-25T19:44:15.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbb9e7fc640 1 -- 172.21.15.29:0/1445977730 learned_addr learned my addr 172.21.15.29:0/1445977730 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:15.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbb9f7fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbba0146640 0x7fbba01e0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34476/0 (socket says 172.21.15.29:34476) 2024-04-25T19:44:15.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbb9effd640 1 --2- 172.21.15.29:0/1445977730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbba006df40 0x7fbba0148a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbb9e7fc640 1 -- 172.21.15.29:0/1445977730 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbba0146640 msgr2=0x7fbba01e0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbb9e7fc640 1 --2- 172.21.15.29:0/1445977730 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbba0146640 0x7fbba01e0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbb9e7fc640 1 -- 172.21.15.29:0/1445977730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbba006df40 msgr2=0x7fbba0148a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbb9e7fc640 1 --2- 172.21.15.29:0/1445977730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbba006df40 0x7fbba0148a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbb9e7fc640 1 -- 172.21.15.29:0/1445977730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba00baaa0 con 0x7fbba00acf70 2024-04-25T19:44:15.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbb9f7fe640 1 --2- 172.21.15.29:0/1445977730 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbba0146640 0x7fbba01e0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:15.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbb9e7fc640 1 --2- 172.21.15.29:0/1445977730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbba00acf70 0x7fbba013a510 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbb9402bfd0 tx=0x7fbb940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbb877fe640 1 -- 172.21.15.29:0/1445977730 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb94034040 con 0x7fbba00acf70 2024-04-25T19:44:15.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbb877fe640 1 -- 172.21.15.29:0/1445977730 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb940443f0 con 0x7fbba00acf70 2024-04-25T19:44:15.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbb877fe640 1 -- 172.21.15.29:0/1445977730 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb940447e0 con 0x7fbba00acf70 2024-04-25T19:44:15.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.363+0000 7fbba5730640 1 -- 172.21.15.29:0/1445977730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbba0147eb0 con 0x7fbba00acf70 2024-04-25T19:44:15.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.364+0000 7fbba5730640 1 -- 172.21.15.29:0/1445977730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbba01457b0 con 0x7fbba00acf70 2024-04-25T19:44:15.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.364+0000 7fbba5730640 1 -- 172.21.15.29:0/1445977730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbba00507b0 con 0x7fbba00acf70 2024-04-25T19:44:15.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.367+0000 7fbb877fe640 1 -- 172.21.15.29:0/1445977730 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7fbb9404b060 con 0x7fbba00acf70 2024-04-25T19:44:15.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.367+0000 7fbb877fe640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:15.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.367+0000 7fbb877fe640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:15.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.367+0000 7fbb877fe640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:15.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.367+0000 7fbb877fe640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:15.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.367+0000 7fbb877fe640 1 --2- 172.21.15.29:0/1445977730 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fbb8c0ac530 0x7fbb8c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.367+0000 7fbb877fe640 1 -- 172.21.15.29:0/1445977730 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7fbb940fb8d0 con 0x7fbba00acf70 2024-04-25T19:44:15.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.367+0000 7fbb9effd640 1 --2- 172.21.15.29:0/1445977730 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fbb8c0ac530 0x7fbb8c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.368+0000 7fbb9effd640 1 --2- 172.21.15.29:0/1445977730 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fbb8c0ac530 0x7fbb8c0aea40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbb98000f60 tx=0x7fbb98009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.370+0000 7fbb877fe640 1 -- 172.21.15.29:0/1445977730 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbb9407fb80 con 0x7fbba00acf70 2024-04-25T19:44:15.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.557+0000 7fbba5730640 1 -- 172.21.15.29:0/1445977730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbba00ba820 con 0x7fbba00acf70 2024-04-25T19:44:15.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.558+0000 7fbb877fe640 1 -- 172.21.15.29:0/1445977730 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+13512 (secure 0 0 0) 0x7fbb9403f070 con 0x7fbba00acf70 2024-04-25T19:44:15.559 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:15.559 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"1b3c39b5-e787-409e-b0bd-f4f491c89d72","created":"2024-04-25T19:43:47.854697+0000","modified":"2024-04-25T19:44:11.035019+0000","last_up_change":"2024-04-25T19:43:57.853237+0000","last_in_change":"2024-04-25T19:43:49.022776+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:43:58.096700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T19:44:03.617011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:44:07.890493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2024-04-25T19:44:08.355570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d805223-ee84-494f-bd31-59091906aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6801","nonce":695704147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6803","nonce":695704147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6807","nonce":695704147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6805","nonce":695704147}]},"public_addr":"172.21.15.29:6801/695704147","cluster_addr":"172.21.15.29:6803/695704147","heartbeat_back_addr":"172.21.15.29:6807/695704147","heartbeat_front_addr":"172.21.15.29:6805/695704147","state":["exists","up"]},{"osd":1,"uuid":"607c5038-38af-440f-8371-e71758549920","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6814","nonce":3135821452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6817","nonce":3135821452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6823","nonce":3135821452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6821","nonce":3135821452}]},"public_addr":"172.21.15.102:6814/3135821452","cluster_addr":"172.21.15.102:6817/3135821452","heartbeat_back_addr":"172.21.15.102:6823/3135821452","heartbeat_front_addr":"172.21.15.102:6821/3135821452","state":["exists","up"]},{"osd":2,"uuid":"6859ae76-54bb-4c55-b0dc-5e2735c0d31c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6801","nonce":2071844144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6803","nonce":2071844144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6811","nonce":2071844144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6806","nonce":2071844144}]},"public_addr":"172.21.15.102:6801/2071844144","cluster_addr":"172.21.15.102:6803/2071844144","heartbeat_back_addr":"172.21.15.102:6811/2071844144","heartbeat_front_addr":"172.21.15.102:6806/2071844144","state":["exists","up"]},{"osd":3,"uuid":"a98b7b88-5206-4f22-84a6-78212e0d5f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6805","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6807","nonce":339944984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6810","nonce":339944984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6820","nonce":339944984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6815","nonce":339944984}]},"public_addr":"172.21.15.102:6807/339944984","cluster_addr":"172.21.15.102:6810/339944984","heartbeat_back_addr":"172.21.15.102:6820/339944984","heartbeat_front_addr":"172.21.15.102:6815/339944984","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.121529+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.201155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.264387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.198713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:44:15.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.560+0000 7fbba5730640 10 mgrc shutdown 2024-04-25T19:44:15.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.560+0000 7fbba5730640 1 -- 172.21.15.29:0/1445977730 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fbb8c0ac530 msgr2=0x7fbb8c0aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.560+0000 7fbba5730640 1 --2- 172.21.15.29:0/1445977730 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fbb8c0ac530 0x7fbb8c0aea40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbb98000f60 tx=0x7fbb98009040 comp rx=0 tx=0).stop 2024-04-25T19:44:15.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.560+0000 7fbba5730640 1 -- 172.21.15.29:0/1445977730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbba00acf70 msgr2=0x7fbba013a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.560+0000 7fbba5730640 1 --2- 172.21.15.29:0/1445977730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbba00acf70 0x7fbba013a510 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbb9402bfd0 tx=0x7fbb940046c0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.560+0000 7fbba5730640 1 -- 172.21.15.29:0/1445977730 shutdown_connections 2024-04-25T19:44:15.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.560+0000 7fbba5730640 1 --2- 172.21.15.29:0/1445977730 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fbb8c0ac530 0x7fbb8c0aea40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.560+0000 7fbba5730640 1 --2- 172.21.15.29:0/1445977730 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fbba0146640 0x7fbba01e0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.560+0000 7fbba5730640 1 --2- 172.21.15.29:0/1445977730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbba00acf70 0x7fbba013a510 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.560+0000 7fbba5730640 1 --2- 172.21.15.29:0/1445977730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbba006df40 0x7fbba0148a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.560+0000 7fbba5730640 1 -- 172.21.15.29:0/1445977730 >> 172.21.15.29:0/1445977730 conn(0x7fbba00ab710 msgr2=0x7fbba00684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:15.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.560+0000 7fbba5730640 1 -- 172.21.15.29:0/1445977730 shutdown_connections 2024-04-25T19:44:15.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.560+0000 7fbba5730640 1 -- 172.21.15.29:0/1445977730 wait complete. 2024-04-25T19:44:15.574 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-25T19:44:15.575 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-25T19:44:15.575 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-25T19:44:15.576 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-25T19:44:15.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.698+0000 7f6c7fe1a640 1 Processor -- start 2024-04-25T19:44:15.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.698+0000 7f26353ec640 1 Processor -- start 2024-04-25T19:44:15.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.698+0000 7f8d7e233640 1 Processor -- start 2024-04-25T19:44:15.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.698+0000 7f6152b1d640 1 Processor -- start 2024-04-25T19:44:15.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.698+0000 7f26353ec640 1 -- start start 2024-04-25T19:44:15.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.698+0000 7f8d7e233640 1 -- start start 2024-04-25T19:44:15.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.698+0000 7f6c7fe1a640 1 -- start start 2024-04-25T19:44:15.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.698+0000 7f6152b1d640 1 -- start start 2024-04-25T19:44:15.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.699+0000 7f26353ec640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f26301442d0 0x7f26300a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.699+0000 7f26353ec640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f263014bc20 0x7f263014e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.699+0000 7f6c7fe1a640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6c780a5290 0x7f6c78143550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.699+0000 7f26353ec640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2630071de0 con 0x7f26300a5f10 2024-04-25T19:44:15.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.699+0000 7f26353ec640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f2630143630 con 0x7f263014bc20 2024-04-25T19:44:15.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.699+0000 7f26353ec640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f2630071b20 con 0x7f26301442d0 2024-04-25T19:44:15.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f6c7fe1a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c7806ecc0 con 0x7f6c78143a90 2024-04-25T19:44:15.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f6c7d3b5640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6c780a5290 0x7f6c78143550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f262f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f263014bc20 0x7f263014e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f262e7fc640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f26300a5f10 0x7f26300a6960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.29:48082/0 (socket says 172.21.15.29:48082) 2024-04-25T19:44:15.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f262effd640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f26301442d0 0x7f26300a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f262e7fc640 1 -- 172.21.15.29:0/2684246382 learned_addr learned my addr 172.21.15.29:0/2684246382 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:15.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f6c7d3b5640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6c780a5290 0x7f6c78143550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34486/0 (socket says 172.21.15.29:34486) 2024-04-25T19:44:15.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f6c7e3b7640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6c78143a90 0x7f6c780a9140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38014/0 (socket says 172.21.15.29:38014) 2024-04-25T19:44:15.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f6c7d3b5640 1 -- 172.21.15.29:0/385575903 learned_addr learned my addr 172.21.15.29:0/385575903 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:15.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f6c7fe1a640 1 -- 172.21.15.29:0/385575903 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c78146830 con 0x7f6c7806e8b0 2024-04-25T19:44:15.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f6c7fe1a640 1 -- 172.21.15.29:0/385575903 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f6c78147480 con 0x7f6c780a5290 2024-04-25T19:44:15.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f8d7e233640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8d780a56d0 0x7f8d780acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f8d7e233640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8d780ad280 0x7f8d781513f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f8d7e233640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d78146c60 con 0x7f8d7806da90 2024-04-25T19:44:15.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f8d7e233640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f8d780ad9d0 con 0x7f8d780ad280 2024-04-25T19:44:15.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f6152b1d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f614c144240 0x7f614c146c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f6152b1d640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f614c147150 0x7f614c151590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f6152b1d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f614c144650 con 0x7f614c143e30 2024-04-25T19:44:15.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f8d7e233640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f8d780ad690 con 0x7f8d780a56d0 2024-04-25T19:44:15.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f262effd640 1 -- 172.21.15.29:0/2684246382 >> v2:172.21.15.29:3300/0 conn(0x7f263014bc20 msgr2=0x7f263014e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:15.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f6152b1d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f614c147560 con 0x7f614c144240 2024-04-25T19:44:15.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f6152b1d640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f614c0a5a90 con 0x7f614c147150 2024-04-25T19:44:15.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f262effd640 1 --2- 172.21.15.29:0/2684246382 >> v2:172.21.15.29:3300/0 conn(0x7f263014bc20 0x7f263014e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f262effd640 1 -- 172.21.15.29:0/2684246382 >> v1:172.21.15.102:6789/0 conn(0x7f26300a5f10 legacy=0x7f26300a6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f8d76ffd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8d780a56d0 0x7f8d780acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f6c7cbb4640 1 -- 172.21.15.29:0/385575903 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2573429288 0 0) 0x7f6c70001490 con 0x7f6c78143a90 2024-04-25T19:44:15.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f8d777fe640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8d7806da90 0x7f8d781472d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.29:48110/0 (socket says 172.21.15.29:48110) 2024-04-25T19:44:15.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f262effd640 1 -- 172.21.15.29:0/2684246382 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26301438d0 con 0x7f26301442d0 2024-04-25T19:44:15.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f8d76ffd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8d780a56d0 0x7f8d780acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34500/0 (socket says 172.21.15.29:34500) 2024-04-25T19:44:15.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f8d76ffd640 1 -- 172.21.15.29:0/1334616944 learned_addr learned my addr 172.21.15.29:0/1334616944 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:15.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f614bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f614c144240 0x7f614c146c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f8d77fff640 1 --2- 172.21.15.29:0/1334616944 >> v2:172.21.15.102:3300/0 conn(0x7f8d780ad280 0x7f8d781513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f6c7cbb4640 1 -- 172.21.15.29:0/385575903 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c64003520 con 0x7f6c78143a90 2024-04-25T19:44:15.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f61508b9640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f614c143e30 0x7f614c0acaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38026/0 (socket says 172.21.15.29:38026) 2024-04-25T19:44:15.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f61508b9640 1 -- 172.21.15.29:0/350953 learned_addr learned my addr 172.21.15.29:0/350953 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:15.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.700+0000 7f61510ba640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f614c147150 0x7f614c151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.701+0000 7f6c7cbb4640 1 -- 172.21.15.29:0/385575903 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 154730565 0 0) 0x7f6c78146830 con 0x7f6c7806e8b0 2024-04-25T19:44:15.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.701+0000 7f6c7cbb4640 1 -- 172.21.15.29:0/385575903 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c70001490 con 0x7f6c7806e8b0 2024-04-25T19:44:15.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.701+0000 7f8d76ffd640 1 -- 172.21.15.29:0/1334616944 >> v2:172.21.15.102:3300/0 conn(0x7f8d780ad280 msgr2=0x7f8d781513f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:15.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.701+0000 7f8d76ffd640 1 --2- 172.21.15.29:0/1334616944 >> v2:172.21.15.102:3300/0 conn(0x7f8d780ad280 0x7f8d781513f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.701+0000 7f8d76ffd640 1 -- 172.21.15.29:0/1334616944 >> v1:172.21.15.102:6789/0 conn(0x7f8d7806da90 legacy=0x7f8d781472d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.701+0000 7f8d76ffd640 1 -- 172.21.15.29:0/1334616944 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d781479e0 con 0x7f8d780a56d0 2024-04-25T19:44:15.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.701+0000 7f61510ba640 1 -- 172.21.15.29:0/350953 >> v2:172.21.15.29:3300/0 conn(0x7f614c144240 msgr2=0x7f614c146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:15.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.701+0000 7f61510ba640 1 --2- 172.21.15.29:0/350953 >> v2:172.21.15.29:3300/0 conn(0x7f614c144240 0x7f614c146c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.701+0000 7f61510ba640 1 -- 172.21.15.29:0/350953 >> v1:172.21.15.29:6789/0 conn(0x7f614c143e30 legacy=0x7f614c0acaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.701+0000 7f61510ba640 1 -- 172.21.15.29:0/350953 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f614c147790 con 0x7f614c147150 2024-04-25T19:44:15.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.701+0000 7f6c7d3b5640 1 -- 172.21.15.29:0/385575903 >> v1:172.21.15.102:6789/0 conn(0x7f6c7806e8b0 legacy=0x7f6c780a1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.701+0000 7f6c7d3b5640 1 -- 172.21.15.29:0/385575903 >> v1:172.21.15.29:6789/0 conn(0x7f6c78143a90 legacy=0x7f6c780a9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.701+0000 7f6c7d3b5640 1 -- 172.21.15.29:0/385575903 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c78146af0 con 0x7f6c780a5290 2024-04-25T19:44:15.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.701+0000 7f262effd640 1 --2- 172.21.15.29:0/2684246382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26301442d0 0x7f26300a59d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2630070780 tx=0x7f262002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c808c61dcfedbab3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.702+0000 7f8d76ffd640 1 --2- 172.21.15.29:0/1334616944 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8d780a56d0 0x7f8d780acd40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c009870 tx=0x7f8d6c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86afa38d4f78c374 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.702+0000 7f262dffb640 1 -- 172.21.15.29:0/2684246382 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2620035020 con 0x7f26301442d0 2024-04-25T19:44:15.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.702+0000 7f8d767fc640 1 -- 172.21.15.29:0/1334616944 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c038070 con 0x7f8d780a56d0 2024-04-25T19:44:15.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.702+0000 7f262dffb640 1 -- 172.21.15.29:0/2684246382 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2620031830 con 0x7f26301442d0 2024-04-25T19:44:15.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.702+0000 7f8d767fc640 1 -- 172.21.15.29:0/1334616944 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d6c031740 con 0x7f8d780a56d0 2024-04-25T19:44:15.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.702+0000 7f8d767fc640 1 -- 172.21.15.29:0/1334616944 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c038070 con 0x7f8d780a56d0 2024-04-25T19:44:15.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.702+0000 7f262dffb640 1 -- 172.21.15.29:0/2684246382 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2620031c00 con 0x7f26301442d0 2024-04-25T19:44:15.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.702+0000 7f6c7d3b5640 1 --2- 172.21.15.29:0/385575903 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6c780a5290 0x7f6c78143550 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6c680098a0 tx=0x7f6c6802dae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93191131590daf3d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.702+0000 7f6c7cbb4640 1 -- 172.21.15.29:0/385575903 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6802f820 con 0x7f6c780a5290 2024-04-25T19:44:15.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.702+0000 7f61510ba640 1 --2- 172.21.15.29:0/350953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f614c147150 0x7f614c151590 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f614c0addd0 tx=0x7f614402be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b721608f0411df5f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.702+0000 7f6c7cbb4640 1 -- 172.21.15.29:0/385575903 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c6802f9c0 con 0x7f6c780a5290 2024-04-25T19:44:15.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.702+0000 7f6c7cbb4640 1 -- 172.21.15.29:0/385575903 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6802fd90 con 0x7f6c780a5290 2024-04-25T19:44:15.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.702+0000 7f614b7fe640 1 -- 172.21.15.29:0/350953 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6144039070 con 0x7f614c147150 2024-04-25T19:44:15.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f614b7fe640 1 -- 172.21.15.29:0/350953 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f614402cbe0 con 0x7f614c147150 2024-04-25T19:44:15.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f614b7fe640 1 -- 172.21.15.29:0/350953 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6144034440 con 0x7f614c147150 2024-04-25T19:44:15.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f26353ec640 1 -- 172.21.15.29:0/2684246382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26301442d0 msgr2=0x7f26300a59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f26353ec640 1 --2- 172.21.15.29:0/2684246382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26301442d0 0x7f26300a59d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2630070780 tx=0x7f262002b8d0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f8d7e233640 1 -- 172.21.15.29:0/1334616944 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8d780a56d0 msgr2=0x7f8d780acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f8d7e233640 1 --2- 172.21.15.29:0/1334616944 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8d780a56d0 0x7f8d780acd40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c009870 tx=0x7f8d6c02b860 comp rx=0 tx=0).stop 2024-04-25T19:44:15.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f26353ec640 1 -- 172.21.15.29:0/2684246382 shutdown_connections 2024-04-25T19:44:15.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f26353ec640 1 --2- 172.21.15.29:0/2684246382 >> v2:172.21.15.29:3300/0 conn(0x7f263014bc20 0x7f263014e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f26353ec640 1 --2- 172.21.15.29:0/2684246382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26301442d0 0x7f26300a59d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f26353ec640 1 -- 172.21.15.29:0/2684246382 >> 172.21.15.29:0/2684246382 conn(0x7f26300b83e0 msgr2=0x7f26300b87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:15.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f26353ec640 1 -- 172.21.15.29:0/2684246382 shutdown_connections 2024-04-25T19:44:15.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f8d7e233640 1 -- 172.21.15.29:0/1334616944 shutdown_connections 2024-04-25T19:44:15.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f8d7e233640 1 --2- 172.21.15.29:0/1334616944 >> v2:172.21.15.102:3300/0 conn(0x7f8d780ad280 0x7f8d781513f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f8d7e233640 1 --2- 172.21.15.29:0/1334616944 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8d780a56d0 0x7f8d780acd40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f8d7e233640 1 -- 172.21.15.29:0/1334616944 >> 172.21.15.29:0/1334616944 conn(0x7f8d780bc3b0 msgr2=0x7f8d780bc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:15.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f8d7e233640 1 -- 172.21.15.29:0/1334616944 shutdown_connections 2024-04-25T19:44:15.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f26353ec640 1 -- 172.21.15.29:0/2684246382 wait complete. 2024-04-25T19:44:15.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f8d7e233640 1 -- 172.21.15.29:0/1334616944 wait complete. 2024-04-25T19:44:15.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f6c7fe1a640 1 -- 172.21.15.29:0/385575903 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6c780a5290 msgr2=0x7f6c78143550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f6c7fe1a640 1 --2- 172.21.15.29:0/385575903 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6c780a5290 0x7f6c78143550 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6c680098a0 tx=0x7f6c6802dae0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f6c7fe1a640 1 -- 172.21.15.29:0/385575903 shutdown_connections 2024-04-25T19:44:15.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f6152b1d640 1 -- 172.21.15.29:0/350953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f614c147150 msgr2=0x7f614c151590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.704+0000 7f6152b1d640 1 --2- 172.21.15.29:0/350953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f614c147150 0x7f614c151590 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f614c0addd0 tx=0x7f614402be30 comp rx=0 tx=0).stop 2024-04-25T19:44:15.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.703+0000 7f6c7fe1a640 1 --2- 172.21.15.29:0/385575903 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6c780a5290 0x7f6c78143550 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.704+0000 7f6c7fe1a640 1 -- 172.21.15.29:0/385575903 >> 172.21.15.29:0/385575903 conn(0x7f6c780a6f10 msgr2=0x7f6c780a7320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:15.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.704+0000 7f6152b1d640 1 -- 172.21.15.29:0/350953 shutdown_connections 2024-04-25T19:44:15.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.704+0000 7f6152b1d640 1 --2- 172.21.15.29:0/350953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f614c147150 0x7f614c151590 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.704+0000 7f6152b1d640 1 --2- 172.21.15.29:0/350953 >> v2:172.21.15.29:3300/0 conn(0x7f614c144240 0x7f614c146c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.704+0000 7f6152b1d640 1 -- 172.21.15.29:0/350953 >> 172.21.15.29:0/350953 conn(0x7f614c0aafc0 msgr2=0x7f614c0ab3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:15.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.704+0000 7f6c7fe1a640 1 -- 172.21.15.29:0/385575903 shutdown_connections 2024-04-25T19:44:15.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.704+0000 7f6c7fe1a640 1 -- 172.21.15.29:0/385575903 wait complete. 2024-04-25T19:44:15.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.704+0000 7f6152b1d640 1 -- 172.21.15.29:0/350953 shutdown_connections 2024-04-25T19:44:15.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.704+0000 7f6152b1d640 1 -- 172.21.15.29:0/350953 wait complete. 2024-04-25T19:44:15.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.705+0000 7f6c7fe1a640 1 Processor -- start 2024-04-25T19:44:15.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.705+0000 7f6152b1d640 1 Processor -- start 2024-04-25T19:44:15.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.705+0000 7f6c7fe1a640 1 -- start start 2024-04-25T19:44:15.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.705+0000 7f6152b1d640 1 -- start start 2024-04-25T19:44:15.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.705+0000 7f8d7e233640 1 Processor -- start 2024-04-25T19:44:15.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.706+0000 7f8d7e233640 1 -- start start 2024-04-25T19:44:15.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.706+0000 7f26353ec640 1 Processor -- start 2024-04-25T19:44:15.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.706+0000 7f26353ec640 1 -- start start 2024-04-25T19:44:15.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.706+0000 7f6c7fe1a640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6c7806e8b0 0x7f6c780b8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.706+0000 7f6c7fe1a640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c780a5290 0x7f6c7809bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.706+0000 7f6c7fe1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c78143a90 0x7f6c781db7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.706+0000 7f6c7fe1a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6c780a5820 con 0x7f6c78143a90 2024-04-25T19:44:15.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.706+0000 7f6152b1d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f614c143e30 0x7f614c141450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.706+0000 7f6152b1d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f614c144240 0x7f614c1e0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.706+0000 7f6c7dbb6640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6c7806e8b0 0x7f6c780b8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.706+0000 7f6c7fe1a640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6c780a56a0 con 0x7f6c780a5290 2024-04-25T19:44:15.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.706+0000 7f6c7fe1a640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f6c780a59a0 con 0x7f6c7806e8b0 2024-04-25T19:44:15.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.706+0000 7f6c7dbb6640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6c7806e8b0 0x7f6c780b8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34506/0 (socket says 172.21.15.29:34506) 2024-04-25T19:44:15.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.706+0000 7f6c7dbb6640 1 -- 172.21.15.29:0/1275791462 learned_addr learned my addr 172.21.15.29:0/1275791462 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:15.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.706+0000 7f6c7e3b7640 1 --2- 172.21.15.29:0/1275791462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c78143a90 0x7f6c781db7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.706+0000 7f6152b1d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f614c147150 0x7f614c0a75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.706+0000 7f6152b1d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f614c0ad330 con 0x7f614c144240 2024-04-25T19:44:15.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.706+0000 7f6152b1d640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f614c0ad1b0 con 0x7f614c143e30 2024-04-25T19:44:15.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.706+0000 7f6152b1d640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f614c151ad0 con 0x7f614c147150 2024-04-25T19:44:15.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.707+0000 7f6c7d3b5640 1 --2- 172.21.15.29:0/1275791462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c780a5290 0x7f6c7809bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.707+0000 7f61508b9640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f614c143e30 0x7f614c141450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.707+0000 7f61510ba640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f614c147150 0x7f614c0a75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.707+0000 7f61510ba640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f614c147150 0x7f614c0a75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34508/0 (socket says 172.21.15.29:34508) 2024-04-25T19:44:15.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.707+0000 7f61510ba640 1 -- 172.21.15.29:0/427424399 learned_addr learned my addr 172.21.15.29:0/427424399 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:15.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.707+0000 7f614bfff640 1 --2- 172.21.15.29:0/427424399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f614c144240 0x7f614c1e0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.707+0000 7f6c7dbb6640 1 -- 172.21.15.29:0/1275791462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c780a5290 msgr2=0x7f6c7809bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.707+0000 7f6c7dbb6640 1 --2- 172.21.15.29:0/1275791462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c780a5290 0x7f6c7809bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.707+0000 7f6c7dbb6640 1 -- 172.21.15.29:0/1275791462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c78143a90 msgr2=0x7f6c781db7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.707+0000 7f6c7dbb6640 1 --2- 172.21.15.29:0/1275791462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c78143a90 0x7f6c781db7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.707+0000 7f6c7dbb6640 1 -- 172.21.15.29:0/1275791462 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c780b77c0 con 0x7f6c7806e8b0 2024-04-25T19:44:15.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.707+0000 7f6c7d3b5640 1 --2- 172.21.15.29:0/1275791462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c780a5290 0x7f6c7809bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:15.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.707+0000 7f61508b9640 1 -- 172.21.15.29:0/427424399 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f614c147150 msgr2=0x7f614c0a75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.707+0000 7f61508b9640 1 --2- 172.21.15.29:0/427424399 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f614c147150 0x7f614c0a75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.707+0000 7f61508b9640 1 -- 172.21.15.29:0/427424399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f614c144240 msgr2=0x7f614c1e0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.707+0000 7f61508b9640 1 --2- 172.21.15.29:0/427424399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f614c144240 0x7f614c1e0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.707+0000 7f61508b9640 1 -- 172.21.15.29:0/427424399 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f614c13dfc0 con 0x7f614c143e30 2024-04-25T19:44:15.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.707+0000 7f61510ba640 1 --2- 172.21.15.29:0/427424399 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f614c147150 0x7f614c0a75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:15.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.707+0000 7f6c7dbb6640 1 --2- 172.21.15.29:0/1275791462 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6c7806e8b0 0x7f6c780b8520 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6c7400c400 tx=0x7f6c7400c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f61508b9640 1 --2- 172.21.15.29:0/427424399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f614c143e30 0x7f614c141450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f614000c440 tx=0x7f614000c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f8d7e233640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d7806da90 0x7f8d7814ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f8d7e233640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8d780a56d0 0x7f8d781e1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f6c6e7fc640 1 -- 172.21.15.29:0/1275791462 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c74017070 con 0x7f6c7806e8b0 2024-04-25T19:44:15.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f8d777fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d7806da90 0x7f8d7814ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f6c6e7fc640 1 -- 172.21.15.29:0/1275791462 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c740052f0 con 0x7f6c7806e8b0 2024-04-25T19:44:15.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f6c6e7fc640 1 -- 172.21.15.29:0/1275791462 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c74003830 con 0x7f6c7806e8b0 2024-04-25T19:44:15.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f8d777fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d7806da90 0x7f8d7814ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36582/0 (socket says 172.21.15.29:36582) 2024-04-25T19:44:15.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f61497fa640 1 -- 172.21.15.29:0/427424399 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6140017070 con 0x7f614c143e30 2024-04-25T19:44:15.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f61497fa640 1 -- 172.21.15.29:0/427424399 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6140005ce0 con 0x7f614c143e30 2024-04-25T19:44:15.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f8d76ffd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8d780a56d0 0x7f8d781e1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f6c7fe1a640 1 -- 172.21.15.29:0/1275791462 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c7809ca50 con 0x7f6c7806e8b0 2024-04-25T19:44:15.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f8d76ffd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8d780a56d0 0x7f8d781e1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34522/0 (socket says 172.21.15.29:34522) 2024-04-25T19:44:15.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f8d777fe640 1 -- 172.21.15.29:0/1135978246 learned_addr learned my addr 172.21.15.29:0/1135978246 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:15.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f8d7e233640 1 --2- 172.21.15.29:0/1135978246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d780ad280 0x7f8d78143e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f6c7fe1a640 1 -- 172.21.15.29:0/1275791462 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c7809d150 con 0x7f6c7806e8b0 2024-04-25T19:44:15.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f6152b1d640 1 -- 172.21.15.29:0/427424399 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f614c13f8d0 con 0x7f614c143e30 2024-04-25T19:44:15.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f8d7e233640 1 -- 172.21.15.29:0/1135978246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8d78151ab0 con 0x7f8d7806da90 2024-04-25T19:44:15.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f8d7e233640 1 -- 172.21.15.29:0/1135978246 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8d78151930 con 0x7f8d780ad280 2024-04-25T19:44:15.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f8d77fff640 1 --2- 172.21.15.29:0/1135978246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d780ad280 0x7f8d78143e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f6152b1d640 1 -- 172.21.15.29:0/427424399 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f614c1401d0 con 0x7f614c143e30 2024-04-25T19:44:15.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f8d7e233640 1 -- 172.21.15.29:0/1135978246 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f8d78151c30 con 0x7f8d780a56d0 2024-04-25T19:44:15.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.708+0000 7f61497fa640 1 -- 172.21.15.29:0/427424399 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61400072d0 con 0x7f614c143e30 2024-04-25T19:44:15.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.709+0000 7f26353ec640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26300a5f10 0x7f263014f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.709+0000 7f26353ec640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f26301442d0 0x7f263009b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.709+0000 7f262effd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26300a5f10 0x7f263014f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.709+0000 7f262effd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26300a5f10 0x7f263014f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.29:49192/0 (socket says 172.21.15.29:49192) 2024-04-25T19:44:15.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.709+0000 7f262effd640 1 -- 172.21.15.29:0/2121461325 learned_addr learned my addr 172.21.15.29:0/2121461325 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:15.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.709+0000 7f262e7fc640 1 --2- 172.21.15.29:0/2121461325 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f26301442d0 0x7f263009b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.709+0000 7f8d777fe640 1 -- 172.21.15.29:0/1135978246 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8d780a56d0 msgr2=0x7f8d781e1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.709+0000 7f26353ec640 1 --2- 172.21.15.29:0/2121461325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f263014bc20 0x7f263009bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.709+0000 7f8d777fe640 1 --2- 172.21.15.29:0/1135978246 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8d780a56d0 0x7f8d781e1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.709+0000 7f26353ec640 1 -- 172.21.15.29:0/2121461325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f26300a71f0 con 0x7f263014bc20 2024-04-25T19:44:15.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.709+0000 7f26353ec640 1 -- 172.21.15.29:0/2121461325 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f26300a7070 con 0x7f26300a5f10 2024-04-25T19:44:15.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.709+0000 7f26353ec640 1 -- 172.21.15.29:0/2121461325 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f2630143210 con 0x7f26301442d0 2024-04-25T19:44:15.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.709+0000 7f8d777fe640 1 -- 172.21.15.29:0/1135978246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d780ad280 msgr2=0x7f8d78143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.709+0000 7f8d777fe640 1 --2- 172.21.15.29:0/1135978246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d780ad280 0x7f8d78143e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.709+0000 7f6c5bfff640 1 -- 172.21.15.29:0/1275791462 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6c780507b0 con 0x7f6c7806e8b0 2024-04-25T19:44:15.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.709+0000 7f8d777fe640 1 -- 172.21.15.29:0/1135978246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d781e1a20 con 0x7f8d7806da90 2024-04-25T19:44:15.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.709+0000 7f8d77fff640 1 --2- 172.21.15.29:0/1135978246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d780ad280 0x7f8d78143e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:44:15.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.709+0000 7f8d76ffd640 1 --2- 172.21.15.29:0/1135978246 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8d780a56d0 0x7f8d781e1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:15.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.709+0000 7f6152b1d640 1 -- 172.21.15.29:0/427424399 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f612c000f80 con 0x7f614c143e30 2024-04-25T19:44:15.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.709+0000 7f262f7fe640 1 --2- 172.21.15.29:0/2121461325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f263014bc20 0x7f263009bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.710+0000 7f8d777fe640 1 --2- 172.21.15.29:0/1135978246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d7806da90 0x7f8d7814ab70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8d6800c620 tx=0x7f8d6800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.710+0000 7f262effd640 1 -- 172.21.15.29:0/2121461325 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f26301442d0 msgr2=0x7f263009b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.710+0000 7f262effd640 1 --2- 172.21.15.29:0/2121461325 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f26301442d0 0x7f263009b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.710+0000 7f262effd640 1 -- 172.21.15.29:0/2121461325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f263014bc20 msgr2=0x7f263009bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.710+0000 7f262effd640 1 --2- 172.21.15.29:0/2121461325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f263014bc20 0x7f263009bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.710+0000 7f262effd640 1 -- 172.21.15.29:0/2121461325 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263009c210 con 0x7f26300a5f10 2024-04-25T19:44:15.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.710+0000 7f262effd640 1 --2- 172.21.15.29:0/2121461325 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26300a5f10 0x7f263014f6c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f262002b910 tx=0x7f26200046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.710+0000 7f8d5ffff640 1 -- 172.21.15.29:0/1135978246 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d68018070 con 0x7f8d7806da90 2024-04-25T19:44:15.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.710+0000 7f26177fe640 1 -- 172.21.15.29:0/2121461325 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2620035050 con 0x7f26300a5f10 2024-04-25T19:44:15.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.710+0000 7f8d7e233640 1 -- 172.21.15.29:0/1135978246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d7814b320 con 0x7f8d7806da90 2024-04-25T19:44:15.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.710+0000 7f8d5ffff640 1 -- 172.21.15.29:0/1135978246 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d680074e0 con 0x7f8d7806da90 2024-04-25T19:44:15.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.710+0000 7f8d7e233640 1 -- 172.21.15.29:0/1135978246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d78144370 con 0x7f8d7806da90 2024-04-25T19:44:15.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.710+0000 7f26177fe640 1 -- 172.21.15.29:0/2121461325 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f262003ea40 con 0x7f26300a5f10 2024-04-25T19:44:15.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.710+0000 7f26177fe640 1 -- 172.21.15.29:0/2121461325 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2620031c00 con 0x7f26300a5f10 2024-04-25T19:44:15.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.711+0000 7f26353ec640 1 -- 172.21.15.29:0/2121461325 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f263014fc00 con 0x7f26300a5f10 2024-04-25T19:44:15.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.711+0000 7f8d5dffb640 1 -- 172.21.15.29:0/1135978246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8d50000f80 con 0x7f8d7806da90 2024-04-25T19:44:15.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.711+0000 7f26353ec640 1 -- 172.21.15.29:0/2121461325 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f263014eaf0 con 0x7f26300a5f10 2024-04-25T19:44:15.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.711+0000 7f8d5ffff640 1 -- 172.21.15.29:0/1135978246 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d68018070 con 0x7f8d7806da90 2024-04-25T19:44:15.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.711+0000 7f61497fa640 1 -- 172.21.15.29:0/427424399 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f614001a070 con 0x7f614c143e30 2024-04-25T19:44:15.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.711+0000 7f61497fa640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:15.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.712+0000 7f6c6e7fc640 1 -- 172.21.15.29:0/1275791462 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f6c7401a070 con 0x7f6c7806e8b0 2024-04-25T19:44:15.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.712+0000 7f6c6e7fc640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:15.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.712+0000 7f26353ec640 1 -- 172.21.15.29:0/2121461325 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f26300507b0 con 0x7f26300a5f10 2024-04-25T19:44:15.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.712+0000 7f61497fa640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:15.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.712+0000 7f61497fa640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:15.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.712+0000 7f61497fa640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:15.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.712+0000 7f61497fa640 1 --2- 172.21.15.29:0/427424399 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f61300ac530 0x7f61300aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.713+0000 7f6c6e7fc640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:15.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.713+0000 7f6c6e7fc640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:15.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.713+0000 7f6c6e7fc640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:15.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.713+0000 7f6c6e7fc640 1 --2- 172.21.15.29:0/1275791462 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f6c5c0ac530 0x7f6c5c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.713+0000 7f614bfff640 1 --2- 172.21.15.29:0/427424399 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f61300ac530 0x7f61300aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.713+0000 7f61497fa640 1 -- 172.21.15.29:0/427424399 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f61400d84a0 con 0x7f614c143e30 2024-04-25T19:44:15.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.713+0000 7f6c7d3b5640 1 --2- 172.21.15.29:0/1275791462 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f6c5c0ac530 0x7f6c5c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.713+0000 7f6c6e7fc640 1 -- 172.21.15.29:0/1275791462 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f6c740d8e90 con 0x7f6c7806e8b0 2024-04-25T19:44:15.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.713+0000 7f61497fa640 1 --2- 172.21.15.29:0/427424399 >> [v2:172.21.15.102:6805/339944984,v1:172.21.15.102:6807/339944984] conn(0x7f61300b40e0 0x7f61300b6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.713+0000 7f61497fa640 1 -- 172.21.15.29:0/427424399 --> [v2:172.21.15.102:6805/339944984,v1:172.21.15.102:6807/339944984] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f61300b6c40 con 0x7f61300b40e0 2024-04-25T19:44:15.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.713+0000 7f61497fa640 1 -- 172.21.15.29:0/427424399 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f61400df050 con 0x7f614c143e30 2024-04-25T19:44:15.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.713+0000 7f6c6e7fc640 1 --2- 172.21.15.29:0/1275791462 >> [v2:172.21.15.29:6800/695704147,v1:172.21.15.29:6801/695704147] conn(0x7f6c5c0b40f0 0x7f6c5c0b6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.713+0000 7f6c6e7fc640 1 -- 172.21.15.29:0/1275791462 --> [v2:172.21.15.29:6800/695704147,v1:172.21.15.29:6801/695704147] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6c740054a0 con 0x7f6c5c0b40f0 2024-04-25T19:44:15.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.713+0000 7f6c6e7fc640 1 -- 172.21.15.29:0/1275791462 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f6c740d9240 con 0x7f6c7806e8b0 2024-04-25T19:44:15.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.713+0000 7f61510ba640 1 --2- 172.21.15.29:0/427424399 >> [v2:172.21.15.102:6805/339944984,v1:172.21.15.102:6807/339944984] conn(0x7f61300b40e0 0x7f61300b6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.713+0000 7f6c7e3b7640 1 --2- 172.21.15.29:0/1275791462 >> [v2:172.21.15.29:6800/695704147,v1:172.21.15.29:6801/695704147] conn(0x7f6c5c0b40f0 0x7f6c5c0b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.714+0000 7f6c7d3b5640 1 --2- 172.21.15.29:0/1275791462 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f6c5c0ac530 0x7f6c5c0aea40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6c68038000 tx=0x7f6c68008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.714+0000 7f6c7e3b7640 1 --2- 172.21.15.29:0/1275791462 >> [v2:172.21.15.29:6800/695704147,v1:172.21.15.29:6801/695704147] conn(0x7f6c5c0b40f0 0x7f6c5c0b6580 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.714+0000 7f61510ba640 1 --2- 172.21.15.29:0/427424399 >> [v2:172.21.15.102:6805/339944984,v1:172.21.15.102:6807/339944984] conn(0x7f61300b40e0 0x7f61300b6570 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.714+0000 7f26177fe640 1 -- 172.21.15.29:0/2121461325 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f2620038020 con 0x7f26300a5f10 2024-04-25T19:44:15.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.714+0000 7f26177fe640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:15.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.714+0000 7f6c6e7fc640 1 -- 172.21.15.29:0/1275791462 <== osd.0 v2:172.21.15.29:6800/695704147 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6c740054a0 con 0x7f6c5c0b40f0 2024-04-25T19:44:15.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.715+0000 7f8d5ffff640 1 -- 172.21.15.29:0/1135978246 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f8d680047e0 con 0x7f8d7806da90 2024-04-25T19:44:15.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.715+0000 7f8d5ffff640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:15.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.715+0000 7f61497fa640 1 -- 172.21.15.29:0/427424399 <== osd.3 v2:172.21.15.102:6805/339944984 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f61300b6c40 con 0x7f61300b40e0 2024-04-25T19:44:15.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.715+0000 7f26177fe640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:15.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.715+0000 7f26177fe640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:15.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.715+0000 7f26177fe640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:15.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.715+0000 7f26177fe640 1 --2- 172.21.15.29:0/2121461325 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f261c0ac530 0x7f261c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.715+0000 7f26177fe640 1 -- 172.21.15.29:0/2121461325 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f26200ffbc0 con 0x7f26300a5f10 2024-04-25T19:44:15.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.715+0000 7f26177fe640 1 --2- 172.21.15.29:0/2121461325 >> [v2:172.21.15.102:6800/2071844144,v1:172.21.15.102:6801/2071844144] conn(0x7f261c0b40e0 0x7f261c0b6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.715+0000 7f26177fe640 1 -- 172.21.15.29:0/2121461325 --> [v2:172.21.15.102:6800/2071844144,v1:172.21.15.102:6801/2071844144] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f261c0b6c40 con 0x7f261c0b40e0 2024-04-25T19:44:15.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.715+0000 7f26177fe640 1 -- 172.21.15.29:0/2121461325 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f2620100050 con 0x7f26300a5f10 2024-04-25T19:44:15.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.716+0000 7f262f7fe640 1 --2- 172.21.15.29:0/2121461325 >> [v2:172.21.15.102:6800/2071844144,v1:172.21.15.102:6801/2071844144] conn(0x7f261c0b40e0 0x7f261c0b6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.716+0000 7f8d5ffff640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:15.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.716+0000 7f8d5ffff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:15.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.716+0000 7f8d5ffff640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:15.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.716+0000 7f8d5ffff640 1 --2- 172.21.15.29:0/1135978246 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f8d640ac6b0 0x7f8d640aebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.716+0000 7f8d5ffff640 1 -- 172.21.15.29:0/1135978246 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f8d680d8630 con 0x7f8d7806da90 2024-04-25T19:44:15.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.716+0000 7f8d5ffff640 1 --2- 172.21.15.29:0/1135978246 >> [v2:172.21.15.102:6812/3135821452,v1:172.21.15.102:6814/3135821452] conn(0x7f8d640b4260 0x7f8d640b66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.716+0000 7f262f7fe640 1 --2- 172.21.15.29:0/2121461325 >> [v2:172.21.15.102:6800/2071844144,v1:172.21.15.102:6801/2071844144] conn(0x7f261c0b40e0 0x7f261c0b6570 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.716+0000 7f8d5ffff640 1 -- 172.21.15.29:0/1135978246 --> [v2:172.21.15.102:6812/3135821452,v1:172.21.15.102:6814/3135821452] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8d68017ec0 con 0x7f8d640b4260 2024-04-25T19:44:15.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.716+0000 7f8d5ffff640 1 -- 172.21.15.29:0/1135978246 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f8d680d8ac0 con 0x7f8d7806da90 2024-04-25T19:44:15.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.716+0000 7f8d77fff640 1 --2- 172.21.15.29:0/1135978246 >> [v2:172.21.15.102:6812/3135821452,v1:172.21.15.102:6814/3135821452] conn(0x7f8d640b4260 0x7f8d640b66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.717+0000 7f8d77fff640 1 --2- 172.21.15.29:0/1135978246 >> [v2:172.21.15.102:6812/3135821452,v1:172.21.15.102:6814/3135821452] conn(0x7f8d640b4260 0x7f8d640b66f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.717+0000 7f26177fe640 1 -- 172.21.15.29:0/2121461325 <== osd.2 v2:172.21.15.102:6800/2071844144 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f261c0b6c40 con 0x7f261c0b40e0 2024-04-25T19:44:15.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.717+0000 7f8d5ffff640 1 -- 172.21.15.29:0/1135978246 <== osd.1 v2:172.21.15.102:6812/3135821452 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8d68017ec0 con 0x7f8d640b4260 2024-04-25T19:44:15.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.722+0000 7f8d76ffd640 1 --2- 172.21.15.29:0/1135978246 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f8d640ac6b0 0x7f8d640aebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.722+0000 7f262e7fc640 1 --2- 172.21.15.29:0/2121461325 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f261c0ac530 0x7f261c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.723+0000 7f614bfff640 1 --2- 172.21.15.29:0/427424399 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f61300ac530 0x7f61300aea40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f614c13ad20 tx=0x7f6138009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.723+0000 7f262e7fc640 1 --2- 172.21.15.29:0/2121461325 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f261c0ac530 0x7f261c0aea40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2624000f60 tx=0x7f2624009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.723+0000 7f8d76ffd640 1 --2- 172.21.15.29:0/1135978246 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f8d640ac6b0 0x7f8d640aebc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c035000 tx=0x7f8d6c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.730+0000 7f6c5bfff640 1 -- 172.21.15.29:0/1275791462 --> [v2:172.21.15.29:6800/695704147,v1:172.21.15.29:6801/695704147] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6c781dbf30 con 0x7f6c5c0b40f0 2024-04-25T19:44:15.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.730+0000 7f6152b1d640 1 -- 172.21.15.29:0/427424399 --> [v2:172.21.15.102:6805/339944984,v1:172.21.15.102:6807/339944984] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f612c002c70 con 0x7f61300b40e0 2024-04-25T19:44:15.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.730+0000 7f6c6e7fc640 1 -- 172.21.15.29:0/1275791462 <== osd.0 v2:172.21.15.29:6800/695704147 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6c781dbf30 con 0x7f6c5c0b40f0 2024-04-25T19:44:15.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.730+0000 7f6c5bfff640 1 -- 172.21.15.29:0/1275791462 >> [v2:172.21.15.29:6800/695704147,v1:172.21.15.29:6801/695704147] conn(0x7f6c5c0b40f0 msgr2=0x7f6c5c0b6580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.730+0000 7f6c5bfff640 1 --2- 172.21.15.29:0/1275791462 >> [v2:172.21.15.29:6800/695704147,v1:172.21.15.29:6801/695704147] conn(0x7f6c5c0b40f0 0x7f6c5c0b6580 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.731+0000 7f6c5bfff640 10 mgrc shutdown 2024-04-25T19:44:15.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.731+0000 7f61497fa640 1 -- 172.21.15.29:0/427424399 <== osd.3 v2:172.21.15.102:6805/339944984 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f612c002c70 con 0x7f61300b40e0 2024-04-25T19:44:15.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.731+0000 7f6c5bfff640 1 -- 172.21.15.29:0/1275791462 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f6c5c0ac530 msgr2=0x7f6c5c0aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.731+0000 7f6c5bfff640 1 --2- 172.21.15.29:0/1275791462 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f6c5c0ac530 0x7f6c5c0aea40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6c68038000 tx=0x7f6c68008030 comp rx=0 tx=0).stop 2024-04-25T19:44:15.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.731+0000 7f6c5bfff640 1 -- 172.21.15.29:0/1275791462 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6c7806e8b0 msgr2=0x7f6c780b8520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.731+0000 7f6c5bfff640 1 --2- 172.21.15.29:0/1275791462 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6c7806e8b0 0x7f6c780b8520 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6c7400c400 tx=0x7f6c7400c8c0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.731+0000 7f6c7e3b7640 1 -- 172.21.15.29:0/1275791462 reap_dead start 2024-04-25T19:44:15.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.731+0000 7f6c5bfff640 1 -- 172.21.15.29:0/1275791462 shutdown_connections 2024-04-25T19:44:15.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.731+0000 7f6c5bfff640 1 -- 172.21.15.29:0/1275791462 >> 172.21.15.29:0/1275791462 conn(0x7f6c780a6f10 msgr2=0x7f6c780a37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:15.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.731+0000 7f6c5bfff640 1 -- 172.21.15.29:0/1275791462 shutdown_connections 2024-04-25T19:44:15.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.731+0000 7f6c5bfff640 1 -- 172.21.15.29:0/1275791462 wait complete. 2024-04-25T19:44:15.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.731+0000 7f6152b1d640 1 -- 172.21.15.29:0/427424399 >> [v2:172.21.15.102:6805/339944984,v1:172.21.15.102:6807/339944984] conn(0x7f61300b40e0 msgr2=0x7f61300b6570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.731+0000 7f6152b1d640 1 --2- 172.21.15.29:0/427424399 >> [v2:172.21.15.102:6805/339944984,v1:172.21.15.102:6807/339944984] conn(0x7f61300b40e0 0x7f61300b6570 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.731+0000 7f6152b1d640 10 mgrc shutdown 2024-04-25T19:44:15.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.731+0000 7f6152b1d640 1 -- 172.21.15.29:0/427424399 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f61300ac530 msgr2=0x7f61300aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.731+0000 7f6152b1d640 1 --2- 172.21.15.29:0/427424399 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f61300ac530 0x7f61300aea40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f614c13ad20 tx=0x7f6138009190 comp rx=0 tx=0).stop 2024-04-25T19:44:15.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.731+0000 7f6152b1d640 1 -- 172.21.15.29:0/427424399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f614c143e30 msgr2=0x7f614c141450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.731+0000 7f6152b1d640 1 --2- 172.21.15.29:0/427424399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f614c143e30 0x7f614c141450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f614000c440 tx=0x7f614000c900 comp rx=0 tx=0).stop 2024-04-25T19:44:15.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.731+0000 7f61510ba640 1 -- 172.21.15.29:0/427424399 reap_dead start 2024-04-25T19:44:15.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.731+0000 7f6152b1d640 1 -- 172.21.15.29:0/427424399 shutdown_connections 2024-04-25T19:44:15.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.731+0000 7f6152b1d640 1 -- 172.21.15.29:0/427424399 >> 172.21.15.29:0/427424399 conn(0x7f614c0aafc0 msgr2=0x7f614c143050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:15.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.732+0000 7f6152b1d640 1 -- 172.21.15.29:0/427424399 shutdown_connections 2024-04-25T19:44:15.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.732+0000 7f6152b1d640 1 -- 172.21.15.29:0/427424399 wait complete. 2024-04-25T19:44:15.764 INFO:teuthology.orchestra.run.smithi029.stdout:38654705670 2024-04-25T19:44:15.764 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T19:44:15.765 INFO:teuthology.orchestra.run.smithi029.stdout:42949672967 2024-04-25T19:44:15.765 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T19:44:15.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.750+0000 7f26353ec640 1 -- 172.21.15.29:0/2121461325 --> [v2:172.21.15.102:6800/2071844144,v1:172.21.15.102:6801/2071844144] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f263009c9b0 con 0x7f261c0b40e0 2024-04-25T19:44:15.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.751+0000 7f26177fe640 1 -- 172.21.15.29:0/2121461325 <== osd.2 v2:172.21.15.102:6800/2071844144 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f263009c9b0 con 0x7f261c0b40e0 2024-04-25T19:44:15.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.751+0000 7f8d5dffb640 1 -- 172.21.15.29:0/1135978246 --> [v2:172.21.15.102:6812/3135821452,v1:172.21.15.102:6814/3135821452] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8d50002c70 con 0x7f8d640b4260 2024-04-25T19:44:15.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.751+0000 7f26353ec640 1 -- 172.21.15.29:0/2121461325 >> [v2:172.21.15.102:6800/2071844144,v1:172.21.15.102:6801/2071844144] conn(0x7f261c0b40e0 msgr2=0x7f261c0b6570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.751+0000 7f26353ec640 1 --2- 172.21.15.29:0/2121461325 >> [v2:172.21.15.102:6800/2071844144,v1:172.21.15.102:6801/2071844144] conn(0x7f261c0b40e0 0x7f261c0b6570 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.751+0000 7f26353ec640 10 mgrc shutdown 2024-04-25T19:44:15.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.751+0000 7f26353ec640 1 -- 172.21.15.29:0/2121461325 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f261c0ac530 msgr2=0x7f261c0aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.751+0000 7f26353ec640 1 --2- 172.21.15.29:0/2121461325 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f261c0ac530 0x7f261c0aea40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2624000f60 tx=0x7f2624009040 comp rx=0 tx=0).stop 2024-04-25T19:44:15.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.751+0000 7f26353ec640 1 -- 172.21.15.29:0/2121461325 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26300a5f10 msgr2=0x7f263014f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.751+0000 7f26353ec640 1 --2- 172.21.15.29:0/2121461325 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26300a5f10 0x7f263014f6c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f262002b910 tx=0x7f26200046c0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.751+0000 7f262f7fe640 1 -- 172.21.15.29:0/2121461325 reap_dead start 2024-04-25T19:44:15.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.751+0000 7f26353ec640 1 -- 172.21.15.29:0/2121461325 shutdown_connections 2024-04-25T19:44:15.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.751+0000 7f8d5ffff640 1 -- 172.21.15.29:0/1135978246 <== osd.1 v2:172.21.15.102:6812/3135821452 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8d50002c70 con 0x7f8d640b4260 2024-04-25T19:44:15.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.751+0000 7f26353ec640 1 -- 172.21.15.29:0/2121461325 >> 172.21.15.29:0/2121461325 conn(0x7f26300b83e0 msgr2=0x7f26300684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:15.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.751+0000 7f26353ec640 1 -- 172.21.15.29:0/2121461325 shutdown_connections 2024-04-25T19:44:15.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.751+0000 7f26353ec640 1 -- 172.21.15.29:0/2121461325 wait complete. 2024-04-25T19:44:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.752+0000 7f8d5dffb640 1 -- 172.21.15.29:0/1135978246 >> [v2:172.21.15.102:6812/3135821452,v1:172.21.15.102:6814/3135821452] conn(0x7f8d640b4260 msgr2=0x7f8d640b66f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.752+0000 7f8d5dffb640 1 --2- 172.21.15.29:0/1135978246 >> [v2:172.21.15.102:6812/3135821452,v1:172.21.15.102:6814/3135821452] conn(0x7f8d640b4260 0x7f8d640b66f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.752+0000 7f8d5dffb640 10 mgrc shutdown 2024-04-25T19:44:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.752+0000 7f8d5dffb640 1 -- 172.21.15.29:0/1135978246 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f8d640ac6b0 msgr2=0x7f8d640aebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.752+0000 7f8d5dffb640 1 --2- 172.21.15.29:0/1135978246 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f8d640ac6b0 0x7f8d640aebc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c035000 tx=0x7f8d6c008030 comp rx=0 tx=0).stop 2024-04-25T19:44:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.752+0000 7f8d5dffb640 1 -- 172.21.15.29:0/1135978246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d7806da90 msgr2=0x7f8d7814ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.752+0000 7f8d5dffb640 1 --2- 172.21.15.29:0/1135978246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d7806da90 0x7f8d7814ab70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8d6800c620 tx=0x7f8d6800cae0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.752+0000 7f8d77fff640 1 -- 172.21.15.29:0/1135978246 reap_dead start 2024-04-25T19:44:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.752+0000 7f8d5dffb640 1 -- 172.21.15.29:0/1135978246 shutdown_connections 2024-04-25T19:44:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.752+0000 7f8d5dffb640 1 -- 172.21.15.29:0/1135978246 >> 172.21.15.29:0/1135978246 conn(0x7f8d780bc3b0 msgr2=0x7f8d78141800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.752+0000 7f8d5dffb640 1 -- 172.21.15.29:0/1135978246 shutdown_connections 2024-04-25T19:44:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.752+0000 7f8d5dffb640 1 -- 172.21.15.29:0/1135978246 wait complete. 2024-04-25T19:44:15.771 INFO:teuthology.orchestra.run.smithi029.stdout:42949672967 2024-04-25T19:44:15.772 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T19:44:15.773 INFO:teuthology.orchestra.run.smithi029.stdout:42949672967 2024-04-25T19:44:15.773 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T19:44:15.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.887+0000 7f6358df3640 1 Processor -- start 2024-04-25T19:44:15.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.887+0000 7f6358df3640 1 -- start start 2024-04-25T19:44:15.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.890+0000 7f6358df3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6354075700 0x7f6354146f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.890+0000 7f6358df3640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6354147460 0x7f6354147870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.890+0000 7f6358df3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6354075b10 con 0x7f6354142dc0 2024-04-25T19:44:15.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.890+0000 7f6358df3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f6354147db0 con 0x7f6354075700 2024-04-25T19:44:15.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.890+0000 7f6358df3640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f6354072580 con 0x7f6354147460 2024-04-25T19:44:15.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.890+0000 7f6351d9b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6354147460 0x7f6354147870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.890+0000 7f6351d9b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6354147460 0x7f6354147870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34536/0 (socket says 172.21.15.29:34536) 2024-04-25T19:44:15.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.890+0000 7f6352d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6354142dc0 0x7f63541431d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38040/0 (socket says 172.21.15.29:38040) 2024-04-25T19:44:15.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.890+0000 7f6351d9b640 1 -- 172.21.15.29:0/339277476 learned_addr learned my addr 172.21.15.29:0/339277476 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:15.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.891+0000 7f635259c640 1 --2- 172.21.15.29:0/339277476 >> v2:172.21.15.29:3300/0 conn(0x7f6354075700 0x7f6354146f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.891+0000 7f6351d9b640 1 -- 172.21.15.29:0/339277476 >> v2:172.21.15.29:3300/0 conn(0x7f6354075700 msgr2=0x7f6354146f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:15.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.891+0000 7f6351d9b640 1 --2- 172.21.15.29:0/339277476 >> v2:172.21.15.29:3300/0 conn(0x7f6354075700 0x7f6354146f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.891+0000 7f6351d9b640 1 -- 172.21.15.29:0/339277476 >> v1:172.21.15.29:6789/0 conn(0x7f6354142dc0 legacy=0x7f63541431d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.891+0000 7f635159a640 1 -- 172.21.15.29:0/339277476 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 872873182 0 0) 0x7f634c001490 con 0x7f6354142dc0 2024-04-25T19:44:15.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.891+0000 7f6351d9b640 1 -- 172.21.15.29:0/339277476 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6354144420 con 0x7f6354147460 2024-04-25T19:44:15.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.893+0000 7f6351d9b640 1 --2- 172.21.15.29:0/339277476 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6354147460 0x7f6354147870 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f63480098a0 tx=0x7f634802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a861fd68d11bd6d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.893+0000 7f635159a640 1 -- 172.21.15.29:0/339277476 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6348034070 con 0x7f6354147460 2024-04-25T19:44:15.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.893+0000 7f635159a640 1 -- 172.21.15.29:0/339277476 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6348030840 con 0x7f6354147460 2024-04-25T19:44:15.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.893+0000 7f635159a640 1 -- 172.21.15.29:0/339277476 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6348030c10 con 0x7f6354147460 2024-04-25T19:44:15.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.895+0000 7f6358df3640 1 -- 172.21.15.29:0/339277476 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6354147460 msgr2=0x7f6354147870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.895+0000 7f6358df3640 1 --2- 172.21.15.29:0/339277476 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6354147460 0x7f6354147870 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f63480098a0 tx=0x7f634802b980 comp rx=0 tx=0).stop 2024-04-25T19:44:15.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.895+0000 7f6358df3640 1 -- 172.21.15.29:0/339277476 shutdown_connections 2024-04-25T19:44:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.895+0000 7f6358df3640 1 --2- 172.21.15.29:0/339277476 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6354147460 0x7f6354147870 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.895+0000 7f6358df3640 1 --2- 172.21.15.29:0/339277476 >> v2:172.21.15.29:3300/0 conn(0x7f6354075700 0x7f6354146f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.895+0000 7f6358df3640 1 -- 172.21.15.29:0/339277476 >> 172.21.15.29:0/339277476 conn(0x7f63540b83e0 msgr2=0x7f63540b87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.895+0000 7f6358df3640 1 -- 172.21.15.29:0/339277476 shutdown_connections 2024-04-25T19:44:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.895+0000 7f6358df3640 1 -- 172.21.15.29:0/339277476 wait complete. 2024-04-25T19:44:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.898+0000 7f6358df3640 1 Processor -- start 2024-04-25T19:44:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.898+0000 7f6358df3640 1 -- start start 2024-04-25T19:44:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.900+0000 7f6358df3640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6354075700 0x7f63540b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.900+0000 7f6358df3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6354142dc0 0x7f6354146610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.900+0000 7f6358df3640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6354147460 0x7f635412f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.900+0000 7f6358df3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f63540a71a0 con 0x7f6354142dc0 2024-04-25T19:44:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.900+0000 7f6358df3640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f63541446c0 con 0x7f6354075700 2024-04-25T19:44:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.900+0000 7f6351d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6354142dc0 0x7f6354146610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.900+0000 7f6351d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6354142dc0 0x7f6354146610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36596/0 (socket says 172.21.15.29:36596) 2024-04-25T19:44:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.900+0000 7f6351d9b640 1 -- 172.21.15.29:0/2896571132 learned_addr learned my addr 172.21.15.29:0/2896571132 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.900+0000 7f6352d9d640 1 --2- 172.21.15.29:0/2896571132 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6354147460 0x7f635412f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.900+0000 7f6358df3640 1 -- 172.21.15.29:0/2896571132 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f63540a6ca0 con 0x7f6354147460 2024-04-25T19:44:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.900+0000 7f635259c640 1 --2- 172.21.15.29:0/2896571132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6354075700 0x7f63540b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.901+0000 7f635259c640 1 -- 172.21.15.29:0/2896571132 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6354147460 msgr2=0x7f635412f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.901+0000 7f635259c640 1 --2- 172.21.15.29:0/2896571132 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6354147460 0x7f635412f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.901+0000 7f635259c640 1 -- 172.21.15.29:0/2896571132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6354142dc0 msgr2=0x7f6354146610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.901+0000 7f635259c640 1 --2- 172.21.15.29:0/2896571132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6354142dc0 0x7f6354146610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.901+0000 7f635259c640 1 -- 172.21.15.29:0/2896571132 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6354146f30 con 0x7f6354075700 2024-04-25T19:44:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.901+0000 7f635259c640 1 --2- 172.21.15.29:0/2896571132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6354075700 0x7f63540b2c60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f634400e870 tx=0x7f634400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.901+0000 7f6342ffd640 1 -- 172.21.15.29:0/2896571132 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6344017070 con 0x7f6354075700 2024-04-25T19:44:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.901+0000 7f6342ffd640 1 -- 172.21.15.29:0/2896571132 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f634400ee60 con 0x7f6354075700 2024-04-25T19:44:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.901+0000 7f6342ffd640 1 -- 172.21.15.29:0/2896571132 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634400cb60 con 0x7f6354075700 2024-04-25T19:44:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.901+0000 7f6358df3640 1 -- 172.21.15.29:0/2896571132 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63540a1d80 con 0x7f6354075700 2024-04-25T19:44:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.901+0000 7f6358df3640 1 -- 172.21.15.29:0/2896571132 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63540b41c0 con 0x7f6354075700 2024-04-25T19:44:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.903+0000 7f6358df3640 1 -- 172.21.15.29:0/2896571132 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63540507b0 con 0x7f6354075700 2024-04-25T19:44:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.905+0000 7f6342ffd640 1 -- 172.21.15.29:0/2896571132 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f634401a070 con 0x7f6354075700 2024-04-25T19:44:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.905+0000 7f6342ffd640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.906+0000 7f6342ffd640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.906+0000 7f6342ffd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.906+0000 7f6342ffd640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.906+0000 7f6342ffd640 1 --2- 172.21.15.29:0/2896571132 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f63340ac560 0x7f63340aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.906+0000 7f6342ffd640 1 -- 172.21.15.29:0/2896571132 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f63440d9880 con 0x7f6354075700 2024-04-25T19:44:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.906+0000 7f6351d9b640 1 --2- 172.21.15.29:0/2896571132 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f63340ac560 0x7f63340aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.907+0000 7f6351d9b640 1 --2- 172.21.15.29:0/2896571132 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f63340ac560 0x7f63340aea70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f63540b46c0 tx=0x7f634803b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.909+0000 7f6342ffd640 1 -- 172.21.15.29:0/2896571132 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f63440a3ea0 con 0x7f6354075700 2024-04-25T19:44:15.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.933+0000 7fc2ea9ec640 1 Processor -- start 2024-04-25T19:44:15.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.933+0000 7fc2ea9ec640 1 -- start start 2024-04-25T19:44:15.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.934+0000 7f4e28f86640 1 Processor -- start 2024-04-25T19:44:15.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.934+0000 7facde773640 1 Processor -- start 2024-04-25T19:44:15.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.934+0000 7facde773640 1 -- start start 2024-04-25T19:44:15.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.934+0000 7f4e28f86640 1 -- start start 2024-04-25T19:44:15.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.934+0000 7fc2ea9ec640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc2e414d2f0 0x7fc2e414f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.934+0000 7fc2ea9ec640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2e40a9340 con 0x7fc2e40acce0 2024-04-25T19:44:15.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.934+0000 7fc2ea9ec640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2e40aa490 con 0x7fc2e40a9800 2024-04-25T19:44:15.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.934+0000 7fc2ea9ec640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fc2e40aa750 con 0x7fc2e414d2f0 2024-04-25T19:44:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.935+0000 7fc2e3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc2e40acce0 0x7fc2e40ad0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38054/0 (socket says 172.21.15.29:38054) 2024-04-25T19:44:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.935+0000 7fc2e3fff640 1 -- 172.21.15.29:0/1743689904 learned_addr learned my addr 172.21.15.29:0/1743689904 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.935+0000 7fc2e8f89640 1 --2- 172.21.15.29:0/1743689904 >> v2:172.21.15.102:3300/0 conn(0x7fc2e414d2f0 0x7fc2e414f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.935+0000 7f4e28f86640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4e240aa840 0x7f4e240a9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.935+0000 7f4e28f86640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4e24146d20 0x7f4e240a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.935+0000 7f4e2259c640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4e240aa840 0x7f4e240a9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.935+0000 7fc2e2ffd640 1 -- 172.21.15.29:0/1743689904 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 343372015 0 0) 0x7fc2e40a9340 con 0x7fc2e40acce0 2024-04-25T19:44:15.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.935+0000 7f4e2259c640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4e240aa840 0x7f4e240a9140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.29:49224/0 (socket says 172.21.15.29:49224) 2024-04-25T19:44:15.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.935+0000 7fc2e2ffd640 1 -- 172.21.15.29:0/1743689904 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2cc003520 con 0x7fc2e40acce0 2024-04-25T19:44:15.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.935+0000 7facde773640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7facd80757c0 0x7facd80a1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.935+0000 7f4e2259c640 1 -- 172.21.15.29:0/4242224066 learned_addr learned my addr 172.21.15.29:0/4242224066 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:15.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.935+0000 7f4e28f86640 1 -- 172.21.15.29:0/4242224066 --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e24147390 con 0x7f4e2406ff60 2024-04-25T19:44:15.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.935+0000 7f4e28f86640 1 -- 172.21.15.29:0/4242224066 --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f4e24147130 con 0x7f4e240aa840 2024-04-25T19:44:15.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.935+0000 7f4e28f86640 1 -- 172.21.15.29:0/4242224066 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f4e24147650 con 0x7f4e24146d20 2024-04-25T19:44:15.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.935+0000 7facd7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7facd80a55a0 0x7facd80a6050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38076/0 (socket says 172.21.15.29:38076) 2024-04-25T19:44:15.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.935+0000 7facd77fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7facd80757c0 0x7facd80a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.935+0000 7facd77fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7facd80757c0 0x7facd80a1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34564/0 (socket says 172.21.15.29:34564) 2024-04-25T19:44:15.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.936+0000 7facd7fff640 1 -- 172.21.15.29:0/1597092191 learned_addr learned my addr 172.21.15.29:0/1597092191 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:15.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.936+0000 7facde773640 1 --2- 172.21.15.29:0/1597092191 >> v2:172.21.15.102:3300/0 conn(0x7facd8147750 0x7facd8147b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.936+0000 7f4e2259c640 1 -- 172.21.15.29:0/4242224066 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4e24146d20 msgr2=0x7f4e240a9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:15.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.936+0000 7facde773640 1 -- 172.21.15.29:0/1597092191 --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facd80a6830 con 0x7facd80a55a0 2024-04-25T19:44:15.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.936+0000 7facde773640 1 -- 172.21.15.29:0/1597092191 --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7facd80a5b30 con 0x7facd8147750 2024-04-25T19:44:15.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.936+0000 7facde773640 1 -- 172.21.15.29:0/1597092191 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7facd8152640 con 0x7facd80757c0 2024-04-25T19:44:15.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.936+0000 7f4e2259c640 1 --2- 172.21.15.29:0/4242224066 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4e24146d20 0x7f4e240a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.936+0000 7f4e2259c640 1 -- 172.21.15.29:0/4242224066 >> v1:172.21.15.29:6789/0 conn(0x7f4e2406ff60 legacy=0x7f4e24143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.936+0000 7facdcd10640 1 --2- 172.21.15.29:0/1597092191 >> v2:172.21.15.102:3300/0 conn(0x7facd8147750 0x7facd8147b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.936+0000 7f4e2259c640 1 -- 172.21.15.29:0/4242224066 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e240aae90 con 0x7f4e240aa840 2024-04-25T19:44:15.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.936+0000 7facd6ffd640 1 -- 172.21.15.29:0/1597092191 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3423238053 0 0) 0x7facd80a6830 con 0x7facd80a55a0 2024-04-25T19:44:15.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.936+0000 7facd6ffd640 1 -- 172.21.15.29:0/1597092191 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7facc0003520 con 0x7facd80a55a0 2024-04-25T19:44:15.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.936+0000 7facdcd10640 1 -- 172.21.15.29:0/1597092191 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7facd80757c0 msgr2=0x7facd80a1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:15.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.936+0000 7facdcd10640 1 --2- 172.21.15.29:0/1597092191 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7facd80757c0 0x7facd80a1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.936+0000 7facdcd10640 1 -- 172.21.15.29:0/1597092191 >> v1:172.21.15.29:6789/0 conn(0x7facd80a55a0 legacy=0x7facd80a6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.936+0000 7facdcd10640 1 -- 172.21.15.29:0/1597092191 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facd81527c0 con 0x7facd8147750 2024-04-25T19:44:15.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.935+0000 7fc2e8f89640 1 -- 172.21.15.29:0/1743689904 >> v1:172.21.15.102:6789/0 conn(0x7fc2e40a9800 legacy=0x7fc2e40a9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.935+0000 7fc2e8f89640 1 -- 172.21.15.29:0/1743689904 >> v1:172.21.15.29:6789/0 conn(0x7fc2e40acce0 legacy=0x7fc2e40ad0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.935+0000 7fc2e8f89640 1 -- 172.21.15.29:0/1743689904 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2e4072800 con 0x7fc2e414d2f0 2024-04-25T19:44:15.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.936+0000 7fc2e8f89640 1 --2- 172.21.15.29:0/1743689904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2e414d2f0 0x7fc2e414f710 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc009d90 tx=0x7fc2dc02bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ed5b4ed855cda8d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.937+0000 7f4e2259c640 1 --2- 172.21.15.29:0/4242224066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e240aa840 0x7f4e240a9140 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4e14009870 tx=0x7f4e1402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d4d5ecccfe21708 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.937+0000 7fc2e2ffd640 1 -- 172.21.15.29:0/1743689904 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc039070 con 0x7fc2e414d2f0 2024-04-25T19:44:15.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.937+0000 7f4e21d9b640 1 -- 172.21.15.29:0/4242224066 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e14034070 con 0x7f4e240aa840 2024-04-25T19:44:15.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.937+0000 7fc2e2ffd640 1 -- 172.21.15.29:0/1743689904 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2dc02ccf0 con 0x7fc2e414d2f0 2024-04-25T19:44:15.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.937+0000 7f4e21d9b640 1 -- 172.21.15.29:0/4242224066 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e14030860 con 0x7f4e240aa840 2024-04-25T19:44:15.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.937+0000 7f4e21d9b640 1 -- 172.21.15.29:0/4242224066 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e14030c30 con 0x7f4e240aa840 2024-04-25T19:44:15.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.937+0000 7fc2e2ffd640 1 -- 172.21.15.29:0/1743689904 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc034660 con 0x7fc2e414d2f0 2024-04-25T19:44:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.937+0000 7facdcd10640 1 --2- 172.21.15.29:0/1597092191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7facd8147750 0x7facd8147b60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7facd0009d90 tx=0x7facd002bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8556f0bc08541ea9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.937+0000 7facd6ffd640 1 -- 172.21.15.29:0/1597092191 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd0039070 con 0x7facd8147750 2024-04-25T19:44:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.937+0000 7facd6ffd640 1 -- 172.21.15.29:0/1597092191 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facd002cce0 con 0x7facd8147750 2024-04-25T19:44:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.937+0000 7facd6ffd640 1 -- 172.21.15.29:0/1597092191 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd0034660 con 0x7facd8147750 2024-04-25T19:44:15.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7fc2ea9ec640 1 -- 172.21.15.29:0/1743689904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2e414d2f0 msgr2=0x7fc2e414f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7fc2ea9ec640 1 --2- 172.21.15.29:0/1743689904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2e414d2f0 0x7fc2e414f710 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc009d90 tx=0x7fc2dc02bf40 comp rx=0 tx=0).stop 2024-04-25T19:44:15.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7f4e28f86640 1 -- 172.21.15.29:0/4242224066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e240aa840 msgr2=0x7f4e240a9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7f4e28f86640 1 --2- 172.21.15.29:0/4242224066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e240aa840 0x7f4e240a9140 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4e14009870 tx=0x7f4e1402b9a0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7f4e28f86640 1 -- 172.21.15.29:0/4242224066 shutdown_connections 2024-04-25T19:44:15.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7fc2ea9ec640 1 -- 172.21.15.29:0/1743689904 shutdown_connections 2024-04-25T19:44:15.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7fc2ea9ec640 1 --2- 172.21.15.29:0/1743689904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2e414d2f0 0x7fc2e414f710 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7f4e28f86640 1 --2- 172.21.15.29:0/4242224066 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4e24146d20 0x7f4e240a9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7fc2ea9ec640 1 -- 172.21.15.29:0/1743689904 >> 172.21.15.29:0/1743689904 conn(0x7fc2e406d6a0 msgr2=0x7fc2e406dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:15.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7f4e28f86640 1 --2- 172.21.15.29:0/4242224066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e240aa840 0x7f4e240a9140 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7f4e28f86640 1 -- 172.21.15.29:0/4242224066 >> 172.21.15.29:0/4242224066 conn(0x7f4e240a7d40 msgr2=0x7f4e240a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:15.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7fc2ea9ec640 1 -- 172.21.15.29:0/1743689904 shutdown_connections 2024-04-25T19:44:15.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7f4e28f86640 1 -- 172.21.15.29:0/4242224066 shutdown_connections 2024-04-25T19:44:15.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7facde773640 1 -- 172.21.15.29:0/1597092191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7facd8147750 msgr2=0x7facd8147b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7fc2ea9ec640 1 -- 172.21.15.29:0/1743689904 wait complete. 2024-04-25T19:44:15.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7f4e28f86640 1 -- 172.21.15.29:0/4242224066 wait complete. 2024-04-25T19:44:15.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7facde773640 1 --2- 172.21.15.29:0/1597092191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7facd8147750 0x7facd8147b60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7facd0009d90 tx=0x7facd002bf30 comp rx=0 tx=0).stop 2024-04-25T19:44:15.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7facde773640 1 -- 172.21.15.29:0/1597092191 shutdown_connections 2024-04-25T19:44:15.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7facde773640 1 --2- 172.21.15.29:0/1597092191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7facd8147750 0x7facd8147b60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7facde773640 1 --2- 172.21.15.29:0/1597092191 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7facd80757c0 0x7facd80a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7facde773640 1 -- 172.21.15.29:0/1597092191 >> 172.21.15.29:0/1597092191 conn(0x7facd80b83e0 msgr2=0x7facd80b87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:15.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7facde773640 1 -- 172.21.15.29:0/1597092191 shutdown_connections 2024-04-25T19:44:15.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.938+0000 7facde773640 1 -- 172.21.15.29:0/1597092191 wait complete. 2024-04-25T19:44:15.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.940+0000 7facde773640 1 Processor -- start 2024-04-25T19:44:15.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.940+0000 7facde773640 1 -- start start 2024-04-25T19:44:15.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.941+0000 7fc2ea9ec640 1 Processor -- start 2024-04-25T19:44:15.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.941+0000 7fc2ea9ec640 1 -- start start 2024-04-25T19:44:15.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.941+0000 7f4e28f86640 1 Processor -- start 2024-04-25T19:44:15.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.941+0000 7f4e28f86640 1 -- start start 2024-04-25T19:44:15.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.941+0000 7facde773640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7facd80757c0 0x7facd80a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.941+0000 7facde773640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7facd80a55a0 0x7facd8072820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.942+0000 7facde773640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facd8147750 0x7facd814a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.942+0000 7facde773640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7facd8151a40 con 0x7facd8147750 2024-04-25T19:44:15.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.942+0000 7facde773640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7facd81518c0 con 0x7facd80a55a0 2024-04-25T19:44:15.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.942+0000 7facde773640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7facd8151bc0 con 0x7facd80757c0 2024-04-25T19:44:15.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.942+0000 7facd7fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7facd80757c0 0x7facd80a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.942+0000 7facdcd10640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facd8147750 0x7facd814a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.942+0000 7facd7fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7facd80757c0 0x7facd80a31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34566/0 (socket says 172.21.15.29:34566) 2024-04-25T19:44:15.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.942+0000 7facd7fff640 1 -- 172.21.15.29:0/1355446378 learned_addr learned my addr 172.21.15.29:0/1355446378 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:15.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.942+0000 7facdcd10640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facd8147750 0x7facd814a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36604/0 (socket says 172.21.15.29:36604) 2024-04-25T19:44:15.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.942+0000 7facd77fe640 1 --2- 172.21.15.29:0/1355446378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7facd80a55a0 0x7facd8072820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.942+0000 7facdcd10640 1 -- 172.21.15.29:0/1355446378 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7facd80757c0 msgr2=0x7facd80a31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.942+0000 7facdcd10640 1 --2- 172.21.15.29:0/1355446378 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7facd80757c0 0x7facd80a31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.942+0000 7facdcd10640 1 -- 172.21.15.29:0/1355446378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7facd80a55a0 msgr2=0x7facd8072820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.942+0000 7facdcd10640 1 --2- 172.21.15.29:0/1355446378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7facd80a55a0 0x7facd8072820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.942+0000 7facdcd10640 1 -- 172.21.15.29:0/1355446378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facd8071b20 con 0x7facd8147750 2024-04-25T19:44:15.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.942+0000 7facd7fff640 1 --2- 172.21.15.29:0/1355446378 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7facd80757c0 0x7facd80a31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:15.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.942+0000 7fc2ea9ec640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fc2e40a9800 0x7fc2e409dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7fc2e3fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fc2e40a9800 0x7fc2e409dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7fc2e3fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fc2e40a9800 0x7fc2e409dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34570/0 (socket says 172.21.15.29:34570) 2024-04-25T19:44:15.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7fc2e3fff640 1 -- 172.21.15.29:0/2130381269 learned_addr learned my addr 172.21.15.29:0/2130381269 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:15.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7fc2ea9ec640 1 --2- 172.21.15.29:0/2130381269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2e40acce0 0x7fc2e414c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7fc2ea9ec640 1 --2- 172.21.15.29:0/2130381269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2e414d2f0 0x7fc2e4148090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7fc2ea9ec640 1 -- 172.21.15.29:0/2130381269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e413a280 con 0x7fc2e40acce0 2024-04-25T19:44:15.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7fc2ea9ec640 1 -- 172.21.15.29:0/2130381269 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e413a100 con 0x7fc2e414d2f0 2024-04-25T19:44:15.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7fc2ea9ec640 1 -- 172.21.15.29:0/2130381269 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fc2e406c590 con 0x7fc2e40a9800 2024-04-25T19:44:15.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7fc2e37fe640 1 --2- 172.21.15.29:0/2130381269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2e40acce0 0x7fc2e414c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7f4e28f86640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e2406ff60 0x7f4e240b7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7facdcd10640 1 --2- 172.21.15.29:0/1355446378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facd8147750 0x7facd814a5f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7facd0033000 tx=0x7facd0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7fc2e8f89640 1 --2- 172.21.15.29:0/2130381269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2e414d2f0 0x7fc2e4148090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7f4e28f86640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e240aa840 0x7f4e2409dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7f4e28f86640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4e24146d20 0x7f4e240b89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7facd4ff9640 1 -- 172.21.15.29:0/1355446378 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd0039050 con 0x7facd8147750 2024-04-25T19:44:15.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7f4e2259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e2406ff60 0x7f4e240b7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7f4e1bfff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e240aa840 0x7f4e2409dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7f4e2259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e2406ff60 0x7f4e240b7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36612/0 (socket says 172.21.15.29:36612) 2024-04-25T19:44:15.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7f4e1bfff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e240aa840 0x7f4e2409dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.29:49264/0 (socket says 172.21.15.29:49264) 2024-04-25T19:44:15.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7facd4ff9640 1 -- 172.21.15.29:0/1355446378 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facd0045070 con 0x7facd8147750 2024-04-25T19:44:15.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7facd4ff9640 1 -- 172.21.15.29:0/1355446378 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd0009450 con 0x7facd8147750 2024-04-25T19:44:15.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7facde773640 1 -- 172.21.15.29:0/1355446378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facd8071d40 con 0x7facd8147750 2024-04-25T19:44:15.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7f4e2259c640 1 -- 172.21.15.29:0/1888441436 learned_addr learned my addr 172.21.15.29:0/1888441436 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:15.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.944+0000 7f4e28f86640 1 -- 172.21.15.29:0/1888441436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4e24143e20 con 0x7f4e2406ff60 2024-04-25T19:44:15.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.944+0000 7f4e28f86640 1 -- 172.21.15.29:0/1888441436 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4e24070580 con 0x7f4e240aa840 2024-04-25T19:44:15.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.944+0000 7f4e22d9d640 1 --2- 172.21.15.29:0/1888441436 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4e24146d20 0x7f4e240b89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.944+0000 7f4e28f86640 1 -- 172.21.15.29:0/1888441436 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f4e24143fa0 con 0x7f4e24146d20 2024-04-25T19:44:15.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7fc2e3fff640 1 -- 172.21.15.29:0/2130381269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2e414d2f0 msgr2=0x7fc2e4148090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7fc2e3fff640 1 --2- 172.21.15.29:0/2130381269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2e414d2f0 0x7fc2e4148090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7fc2e3fff640 1 -- 172.21.15.29:0/2130381269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2e40acce0 msgr2=0x7fc2e414c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7fc2e3fff640 1 --2- 172.21.15.29:0/2130381269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2e40acce0 0x7fc2e414c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7fc2e3fff640 1 -- 172.21.15.29:0/2130381269 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2e4153640 con 0x7fc2e40a9800 2024-04-25T19:44:15.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7fc2e37fe640 1 --2- 172.21.15.29:0/2130381269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2e40acce0 0x7fc2e414c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:15.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.944+0000 7f4e2259c640 1 -- 172.21.15.29:0/1888441436 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4e24146d20 msgr2=0x7f4e240b89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.944+0000 7fc2e3fff640 1 --2- 172.21.15.29:0/2130381269 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fc2e40a9800 0x7fc2e409dd20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc2d400c400 tx=0x7fc2d400c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.944+0000 7f4e2259c640 1 --2- 172.21.15.29:0/1888441436 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4e24146d20 0x7f4e240b89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.944+0000 7f4e2259c640 1 -- 172.21.15.29:0/1888441436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e240aa840 msgr2=0x7f4e2409dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:15.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.944+0000 7f4e2259c640 1 --2- 172.21.15.29:0/1888441436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e240aa840 0x7f4e2409dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:15.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.944+0000 7f4e2259c640 1 -- 172.21.15.29:0/1888441436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e2409b360 con 0x7f4e2406ff60 2024-04-25T19:44:15.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.944+0000 7f4e22d9d640 1 --2- 172.21.15.29:0/1888441436 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4e24146d20 0x7f4e240b89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:44:15.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.943+0000 7facde773640 1 -- 172.21.15.29:0/1355446378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facd8148100 con 0x7facd8147750 2024-04-25T19:44:15.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.945+0000 7f4e2259c640 1 --2- 172.21.15.29:0/1888441436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e2406ff60 0x7f4e240b7f90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4e14000c00 tx=0x7f4e140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.945+0000 7fc2e0ff9640 1 -- 172.21.15.29:0/2130381269 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4017070 con 0x7fc2e40a9800 2024-04-25T19:44:15.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.945+0000 7fc2e0ff9640 1 -- 172.21.15.29:0/2130381269 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2d4005ce0 con 0x7fc2e40a9800 2024-04-25T19:44:15.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.945+0000 7fc2ea9ec640 1 -- 172.21.15.29:0/2130381269 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2e409e260 con 0x7fc2e40a9800 2024-04-25T19:44:15.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.945+0000 7facde773640 1 -- 172.21.15.29:0/1355446378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facd8148490 con 0x7facd8147750 2024-04-25T19:44:15.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.945+0000 7fc2e0ff9640 1 -- 172.21.15.29:0/2130381269 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d40072d0 con 0x7fc2e40a9800 2024-04-25T19:44:15.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.945+0000 7fc2ea9ec640 1 -- 172.21.15.29:0/2130381269 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2e412f530 con 0x7fc2e40a9800 2024-04-25T19:44:15.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.945+0000 7f4e1affd640 1 -- 172.21.15.29:0/1888441436 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e14034040 con 0x7f4e2406ff60 2024-04-25T19:44:15.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.945+0000 7f4e1affd640 1 -- 172.21.15.29:0/1888441436 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e140433f0 con 0x7f4e2406ff60 2024-04-25T19:44:15.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.945+0000 7f4e1affd640 1 -- 172.21.15.29:0/1888441436 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e140437e0 con 0x7f4e2406ff60 2024-04-25T19:44:15.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.945+0000 7f4e28f86640 1 -- 172.21.15.29:0/1888441436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e240b84d0 con 0x7f4e2406ff60 2024-04-25T19:44:15.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.946+0000 7f4e28f86640 1 -- 172.21.15.29:0/1888441436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e2409e2f0 con 0x7f4e2406ff60 2024-04-25T19:44:15.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.947+0000 7fc2ea9ec640 1 -- 172.21.15.29:0/2130381269 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2e4149bb0 con 0x7fc2e40a9800 2024-04-25T19:44:15.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.947+0000 7f4e28f86640 1 -- 172.21.15.29:0/1888441436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e240507b0 con 0x7f4e2406ff60 2024-04-25T19:44:15.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.947+0000 7facd4ff9640 1 -- 172.21.15.29:0/1355446378 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7facd003e850 con 0x7facd8147750 2024-04-25T19:44:15.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.947+0000 7facd4ff9640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:15.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.948+0000 7facd4ff9640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:15.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.948+0000 7facd4ff9640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:15.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.948+0000 7facd4ff9640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:15.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.948+0000 7facd4ff9640 1 --2- 172.21.15.29:0/1355446378 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7facc40ac5b0 0x7facc40aeac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.948+0000 7facd4ff9640 1 -- 172.21.15.29:0/1355446378 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7facd00fc850 con 0x7facd8147750 2024-04-25T19:44:15.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.948+0000 7fc2e0ff9640 1 -- 172.21.15.29:0/2130381269 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7fc2d401a070 con 0x7fc2e40a9800 2024-04-25T19:44:15.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.948+0000 7fc2e0ff9640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:15.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.948+0000 7facd7fff640 1 --2- 172.21.15.29:0/1355446378 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7facc40ac5b0 0x7facc40aeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.948+0000 7facd7fff640 1 --2- 172.21.15.29:0/1355446378 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7facc40ac5b0 0x7facc40aeac0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7facc80011e0 tx=0x7facc8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.949+0000 7fc2e0ff9640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:15.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.949+0000 7fc2e0ff9640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:15.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.949+0000 7fc2e0ff9640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:15.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.949+0000 7fc2e0ff9640 1 --2- 172.21.15.29:0/2130381269 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fc2d00ac560 0x7fc2d00aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.949+0000 7fc2e37fe640 1 --2- 172.21.15.29:0/2130381269 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fc2d00ac560 0x7fc2d00aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.949+0000 7fc2e0ff9640 1 -- 172.21.15.29:0/2130381269 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7fc2d40d8650 con 0x7fc2e40a9800 2024-04-25T19:44:15.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.949+0000 7fc2e37fe640 1 --2- 172.21.15.29:0/2130381269 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fc2d00ac560 0x7fc2d00aea70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc2e40aa1d0 tx=0x7fc2d8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.951+0000 7facd4ff9640 1 -- 172.21.15.29:0/1355446378 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7facd00c7420 con 0x7facd8147750 2024-04-25T19:44:15.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.954+0000 7fc2e0ff9640 1 -- 172.21.15.29:0/2130381269 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc2e4149bb0 con 0x7fc2e40a9800 2024-04-25T19:44:15.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.955+0000 7f4e1affd640 1 -- 172.21.15.29:0/1888441436 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f4e1404a060 con 0x7f4e2406ff60 2024-04-25T19:44:15.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.955+0000 7f4e1affd640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:15.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.956+0000 7f4e1affd640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:15.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.956+0000 7f4e1affd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:15.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.956+0000 7f4e1affd640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:15.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.956+0000 7f4e1affd640 1 --2- 172.21.15.29:0/1888441436 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f4e040b0a40 0x7f4e040b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:15.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.956+0000 7f4e1bfff640 1 --2- 172.21.15.29:0/1888441436 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f4e040b0a40 0x7f4e040b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:15.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.956+0000 7f4e1affd640 1 -- 172.21.15.29:0/1888441436 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f4e140fb9d0 con 0x7f4e2406ff60 2024-04-25T19:44:15.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.957+0000 7f4e1bfff640 1 --2- 172.21.15.29:0/1888441436 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f4e040b0a40 0x7f4e040b2f50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4e240a1b70 tx=0x7f4e0c009060 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:15.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:15.958+0000 7f4e1affd640 1 -- 172.21.15.29:0/1888441436 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4e1403e070 con 0x7f4e2406ff60 2024-04-25T19:44:16.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.104+0000 7f6358df3640 1 -- 172.21.15.29:0/2896571132 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f63540b32b0 con 0x7f6354075700 2024-04-25T19:44:16.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.105+0000 7f6342ffd640 1 -- 172.21.15.29:0/2896571132 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f63440a81a0 con 0x7f6354075700 2024-04-25T19:44:16.106 INFO:teuthology.orchestra.run.smithi029.stdout:38654705669 2024-04-25T19:44:16.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.107+0000 7f6358df3640 10 mgrc shutdown 2024-04-25T19:44:16.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.107+0000 7f6358df3640 1 -- 172.21.15.29:0/2896571132 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f63340ac560 msgr2=0x7f63340aea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:16.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.107+0000 7f6358df3640 1 --2- 172.21.15.29:0/2896571132 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f63340ac560 0x7f63340aea70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f63540b46c0 tx=0x7f634803b040 comp rx=0 tx=0).stop 2024-04-25T19:44:16.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.107+0000 7f6358df3640 1 -- 172.21.15.29:0/2896571132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6354075700 msgr2=0x7f63540b2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:16.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.107+0000 7f6358df3640 1 --2- 172.21.15.29:0/2896571132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6354075700 0x7f63540b2c60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f634400e870 tx=0x7f634400ed30 comp rx=0 tx=0).stop 2024-04-25T19:44:16.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.108+0000 7f6358df3640 1 -- 172.21.15.29:0/2896571132 shutdown_connections 2024-04-25T19:44:16.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.108+0000 7f6358df3640 1 --2- 172.21.15.29:0/2896571132 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f63340ac560 0x7f63340aea70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:16.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.108+0000 7f6358df3640 1 --2- 172.21.15.29:0/2896571132 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6354147460 0x7f635412f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:16.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.108+0000 7f6358df3640 1 --2- 172.21.15.29:0/2896571132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6354142dc0 0x7f6354146610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:16.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.108+0000 7f6358df3640 1 --2- 172.21.15.29:0/2896571132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6354075700 0x7f63540b2c60 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f634400e870 tx=0x7f634400ed30 comp rx=0 tx=0).stop 2024-04-25T19:44:16.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.108+0000 7f6358df3640 1 -- 172.21.15.29:0/2896571132 >> 172.21.15.29:0/2896571132 conn(0x7f63540b83e0 msgr2=0x7f63540684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:16.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.108+0000 7f6358df3640 1 -- 172.21.15.29:0/2896571132 shutdown_connections 2024-04-25T19:44:16.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.108+0000 7f6358df3640 1 -- 172.21.15.29:0/2896571132 wait complete. 2024-04-25T19:44:16.124 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.0 2024-04-25T19:44:16.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.151+0000 7fc2ea9ec640 1 -- 172.21.15.29:0/2130381269 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fc2e40507b0 con 0x7fc2e40a9800 2024-04-25T19:44:16.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.151+0000 7fc2e0ff9640 1 -- 172.21.15.29:0/2130381269 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc2d40ab340 con 0x7fc2e40a9800 2024-04-25T19:44:16.152 INFO:teuthology.orchestra.run.smithi029.stdout:42949672966 2024-04-25T19:44:16.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.154+0000 7fc2ea9ec640 10 mgrc shutdown 2024-04-25T19:44:16.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.154+0000 7fc2ea9ec640 1 -- 172.21.15.29:0/2130381269 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fc2d00ac560 msgr2=0x7fc2d00aea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:16.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.154+0000 7fc2ea9ec640 1 --2- 172.21.15.29:0/2130381269 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fc2d00ac560 0x7fc2d00aea70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc2e40aa1d0 tx=0x7fc2d8009040 comp rx=0 tx=0).stop 2024-04-25T19:44:16.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.154+0000 7fc2ea9ec640 1 -- 172.21.15.29:0/2130381269 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fc2e40a9800 msgr2=0x7fc2e409dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:16.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.154+0000 7fc2ea9ec640 1 --2- 172.21.15.29:0/2130381269 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fc2e40a9800 0x7fc2e409dd20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc2d400c400 tx=0x7fc2d400c8c0 comp rx=0 tx=0).stop 2024-04-25T19:44:16.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.154+0000 7fc2ea9ec640 1 -- 172.21.15.29:0/2130381269 shutdown_connections 2024-04-25T19:44:16.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.154+0000 7fc2ea9ec640 1 --2- 172.21.15.29:0/2130381269 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fc2d00ac560 0x7fc2d00aea70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:16.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.154+0000 7fc2ea9ec640 1 --2- 172.21.15.29:0/2130381269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2e414d2f0 0x7fc2e4148090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:16.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.154+0000 7fc2ea9ec640 1 --2- 172.21.15.29:0/2130381269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2e40acce0 0x7fc2e414c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:16.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.154+0000 7fc2ea9ec640 1 --2- 172.21.15.29:0/2130381269 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fc2e40a9800 0x7fc2e409dd20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:16.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.154+0000 7fc2ea9ec640 1 -- 172.21.15.29:0/2130381269 >> 172.21.15.29:0/2130381269 conn(0x7fc2e406d6a0 msgr2=0x7fc2e4145960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:16.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.154+0000 7fc2ea9ec640 1 -- 172.21.15.29:0/2130381269 shutdown_connections 2024-04-25T19:44:16.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.154+0000 7fc2ea9ec640 1 -- 172.21.15.29:0/2130381269 wait complete. 2024-04-25T19:44:16.170 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672967 got 42949672966 for osd.3 2024-04-25T19:44:16.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.171+0000 7facde773640 1 -- 172.21.15.29:0/1355446378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7facd80507b0 con 0x7facd8147750 2024-04-25T19:44:16.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.171+0000 7facd4ff9640 1 -- 172.21.15.29:0/1355446378 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7facd00c6d20 con 0x7facd8147750 2024-04-25T19:44:16.172 INFO:teuthology.orchestra.run.smithi029.stdout:42949672966 2024-04-25T19:44:16.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.173+0000 7facde773640 10 mgrc shutdown 2024-04-25T19:44:16.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.173+0000 7facde773640 1 -- 172.21.15.29:0/1355446378 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7facc40ac5b0 msgr2=0x7facc40aeac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:16.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.173+0000 7facde773640 1 --2- 172.21.15.29:0/1355446378 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7facc40ac5b0 0x7facc40aeac0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7facc80011e0 tx=0x7facc8009040 comp rx=0 tx=0).stop 2024-04-25T19:44:16.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.173+0000 7facde773640 1 -- 172.21.15.29:0/1355446378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facd8147750 msgr2=0x7facd814a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:16.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.173+0000 7facde773640 1 --2- 172.21.15.29:0/1355446378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facd8147750 0x7facd814a5f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7facd0033000 tx=0x7facd0004a40 comp rx=0 tx=0).stop 2024-04-25T19:44:16.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.173+0000 7facde773640 1 -- 172.21.15.29:0/1355446378 shutdown_connections 2024-04-25T19:44:16.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.173+0000 7facde773640 1 --2- 172.21.15.29:0/1355446378 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7facc40ac5b0 0x7facc40aeac0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:16.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.173+0000 7facde773640 1 --2- 172.21.15.29:0/1355446378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facd8147750 0x7facd814a5f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:16.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.173+0000 7facde773640 1 --2- 172.21.15.29:0/1355446378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7facd80a55a0 0x7facd8072820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:16.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.173+0000 7facde773640 1 --2- 172.21.15.29:0/1355446378 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7facd80757c0 0x7facd80a31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:16.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.173+0000 7facde773640 1 -- 172.21.15.29:0/1355446378 >> 172.21.15.29:0/1355446378 conn(0x7facd80b83e0 msgr2=0x7facd80684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:16.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.173+0000 7facde773640 1 -- 172.21.15.29:0/1355446378 shutdown_connections 2024-04-25T19:44:16.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.173+0000 7facde773640 1 -- 172.21.15.29:0/1355446378 wait complete. 2024-04-25T19:44:16.185 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672967 got 42949672966 for osd.2 2024-04-25T19:44:16.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.187+0000 7f4e28f86640 1 -- 172.21.15.29:0/1888441436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f4e240a9960 con 0x7f4e2406ff60 2024-04-25T19:44:16.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.187+0000 7f4e1affd640 1 -- 172.21.15.29:0/1888441436 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4e140c66c0 con 0x7f4e2406ff60 2024-04-25T19:44:16.188 INFO:teuthology.orchestra.run.smithi029.stdout:42949672966 2024-04-25T19:44:16.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.189+0000 7f4e28f86640 10 mgrc shutdown 2024-04-25T19:44:16.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.189+0000 7f4e28f86640 1 -- 172.21.15.29:0/1888441436 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f4e040b0a40 msgr2=0x7f4e040b2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:16.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.189+0000 7f4e28f86640 1 --2- 172.21.15.29:0/1888441436 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f4e040b0a40 0x7f4e040b2f50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4e240a1b70 tx=0x7f4e0c009060 comp rx=0 tx=0).stop 2024-04-25T19:44:16.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.189+0000 7f4e28f86640 1 -- 172.21.15.29:0/1888441436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e2406ff60 msgr2=0x7f4e240b7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:16.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.189+0000 7f4e28f86640 1 --2- 172.21.15.29:0/1888441436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e2406ff60 0x7f4e240b7f90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4e14000c00 tx=0x7f4e140046c0 comp rx=0 tx=0).stop 2024-04-25T19:44:16.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.189+0000 7f4e28f86640 1 -- 172.21.15.29:0/1888441436 shutdown_connections 2024-04-25T19:44:16.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.189+0000 7f4e28f86640 1 --2- 172.21.15.29:0/1888441436 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f4e040b0a40 0x7f4e040b2f50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:16.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.189+0000 7f4e28f86640 1 --2- 172.21.15.29:0/1888441436 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4e24146d20 0x7f4e240b89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:16.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.189+0000 7f4e28f86640 1 --2- 172.21.15.29:0/1888441436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e240aa840 0x7f4e2409dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:16.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.189+0000 7f4e28f86640 1 --2- 172.21.15.29:0/1888441436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e2406ff60 0x7f4e240b7f90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:16.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.189+0000 7f4e28f86640 1 -- 172.21.15.29:0/1888441436 >> 172.21.15.29:0/1888441436 conn(0x7f4e240a7d40 msgr2=0x7f4e24141d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:16.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.190+0000 7f4e28f86640 1 -- 172.21.15.29:0/1888441436 shutdown_connections 2024-04-25T19:44:16.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:16.190+0000 7f4e28f86640 1 -- 172.21.15.29:0/1888441436 wait complete. 2024-04-25T19:44:16.202 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672967 got 42949672966 for osd.1 2024-04-25T19:44:17.125 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T19:44:17.171 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T19:44:17.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T19:44:17.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T19:44:17.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.255+0000 7f65eaaf1640 1 Processor -- start 2024-04-25T19:44:17.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.255+0000 7f65eaaf1640 1 -- start start 2024-04-25T19:44:17.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.256+0000 7f65eaaf1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f65e40ba0e0 0x7f65e40aab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.256+0000 7f65eaaf1640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f65e4146730 0x7f65e40abe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.256+0000 7f65eaaf1640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65e40ac5b0 con 0x7f65e40aba00 2024-04-25T19:44:17.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.256+0000 7f65eaaf1640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f65e40ab570 con 0x7f65e40ba0e0 2024-04-25T19:44:17.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.256+0000 7f65e908e640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f65e4146730 0x7f65e40abe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.256+0000 7f65e888d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f65e40ba0e0 0x7f65e40aab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.257+0000 7f65e908e640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f65e4146730 0x7f65e40abe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34596/0 (socket says 172.21.15.29:34596) 2024-04-25T19:44:17.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.257+0000 7f65e908e640 1 -- 172.21.15.29:0/362941137 learned_addr learned my addr 172.21.15.29:0/362941137 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:17.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.257+0000 7f65eaaf1640 1 -- 172.21.15.29:0/362941137 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f65e414a170 con 0x7f65e4146730 2024-04-25T19:44:17.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.257+0000 7f65e908e640 1 -- 172.21.15.29:0/362941137 >> v2:172.21.15.29:3300/0 conn(0x7f65e40ba0e0 msgr2=0x7f65e40aab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:17.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.257+0000 7f65e908e640 1 --2- 172.21.15.29:0/362941137 >> v2:172.21.15.29:3300/0 conn(0x7f65e40ba0e0 0x7f65e40aab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.257+0000 7f65e908e640 1 -- 172.21.15.29:0/362941137 >> v1:172.21.15.102:6789/0 conn(0x7f65e40aba00 legacy=0x7f65e4146020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.257+0000 7f65e908e640 1 -- 172.21.15.29:0/362941137 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e4152470 con 0x7f65e4146730 2024-04-25T19:44:17.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.257+0000 7f65db7fe640 1 -- 172.21.15.29:0/362941137 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 241413106 0 0) 0x7f65e40ac5b0 con 0x7f65e40aba00 2024-04-25T19:44:17.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.257+0000 7f65e888d640 1 --2- 172.21.15.29:0/362941137 >> v2:172.21.15.29:3300/0 conn(0x7f65e40ba0e0 0x7f65e40aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:17.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.258+0000 7f65e908e640 1 --2- 172.21.15.29:0/362941137 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f65e4146730 0x7f65e40abe10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009e50 tx=0x7f65dc02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8291664e2598600 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:17.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.258+0000 7f65db7fe640 1 -- 172.21.15.29:0/362941137 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65dc02cb00 con 0x7f65e4146730 2024-04-25T19:44:17.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.258+0000 7f65db7fe640 1 -- 172.21.15.29:0/362941137 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65dc02cca0 con 0x7f65e4146730 2024-04-25T19:44:17.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.258+0000 7f65db7fe640 1 -- 172.21.15.29:0/362941137 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65dc034610 con 0x7f65e4146730 2024-04-25T19:44:17.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.259+0000 7f65eaaf1640 1 -- 172.21.15.29:0/362941137 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f65e4146730 msgr2=0x7f65e40abe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.259+0000 7f65eaaf1640 1 --2- 172.21.15.29:0/362941137 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f65e4146730 0x7f65e40abe10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009e50 tx=0x7f65dc02bf60 comp rx=0 tx=0).stop 2024-04-25T19:44:17.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.259+0000 7f65eaaf1640 1 -- 172.21.15.29:0/362941137 shutdown_connections 2024-04-25T19:44:17.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.259+0000 7f65eaaf1640 1 --2- 172.21.15.29:0/362941137 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f65e4146730 0x7f65e40abe10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.259+0000 7f65eaaf1640 1 --2- 172.21.15.29:0/362941137 >> v2:172.21.15.29:3300/0 conn(0x7f65e40ba0e0 0x7f65e40aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.259+0000 7f65eaaf1640 1 -- 172.21.15.29:0/362941137 >> 172.21.15.29:0/362941137 conn(0x7f65e4072ab0 msgr2=0x7f65e40a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:17.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.259+0000 7f65eaaf1640 1 -- 172.21.15.29:0/362941137 shutdown_connections 2024-04-25T19:44:17.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.259+0000 7f65eaaf1640 1 -- 172.21.15.29:0/362941137 wait complete. 2024-04-25T19:44:17.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.262+0000 7f65eaaf1640 1 Processor -- start 2024-04-25T19:44:17.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.262+0000 7f65eaaf1640 1 -- start start 2024-04-25T19:44:17.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.263+0000 7f65eaaf1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65e40aba00 0x7f65e41404d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.263+0000 7f65eaaf1640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f65e40ba0e0 0x7f65e40b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.264+0000 7f65eaaf1640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65e41442b0 0x7f65e4138270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.264+0000 7f65eaaf1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f65e4151830 con 0x7f65e40aba00 2024-04-25T19:44:17.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.264+0000 7f65eaaf1640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f65e41516b0 con 0x7f65e41442b0 2024-04-25T19:44:17.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.264+0000 7f65eaaf1640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f65e41519b0 con 0x7f65e40ba0e0 2024-04-25T19:44:17.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.264+0000 7f65dbfff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f65e40ba0e0 0x7f65e40b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.264+0000 7f65e888d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65e40aba00 0x7f65e41404d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.264+0000 7f65dbfff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f65e40ba0e0 0x7f65e40b4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34612/0 (socket says 172.21.15.29:34612) 2024-04-25T19:44:17.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.264+0000 7f65dbfff640 1 -- 172.21.15.29:0/2683293821 learned_addr learned my addr 172.21.15.29:0/2683293821 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:17.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.264+0000 7f65e908e640 1 --2- 172.21.15.29:0/2683293821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65e41442b0 0x7f65e4138270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.264+0000 7f65dbfff640 1 -- 172.21.15.29:0/2683293821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65e41442b0 msgr2=0x7f65e4138270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.264+0000 7f65dbfff640 1 --2- 172.21.15.29:0/2683293821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65e41442b0 0x7f65e4138270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.264+0000 7f65dbfff640 1 -- 172.21.15.29:0/2683293821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65e40aba00 msgr2=0x7f65e41404d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.264+0000 7f65dbfff640 1 --2- 172.21.15.29:0/2683293821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65e40aba00 0x7f65e41404d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.264+0000 7f65dbfff640 1 -- 172.21.15.29:0/2683293821 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e4140a10 con 0x7f65e40ba0e0 2024-04-25T19:44:17.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.264+0000 7f65e908e640 1 --2- 172.21.15.29:0/2683293821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65e41442b0 0x7f65e4138270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:17.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.264+0000 7f65e888d640 1 --2- 172.21.15.29:0/2683293821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65e40aba00 0x7f65e41404d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:17.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.265+0000 7f65dbfff640 1 --2- 172.21.15.29:0/2683293821 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f65e40ba0e0 0x7f65e40b4b00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f65d400c3c0 tx=0x7f65d400c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:17.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.265+0000 7f65d97fa640 1 -- 172.21.15.29:0/2683293821 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d4017070 con 0x7f65e40ba0e0 2024-04-25T19:44:17.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.265+0000 7f65d97fa640 1 -- 172.21.15.29:0/2683293821 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65d40035f0 con 0x7f65e40ba0e0 2024-04-25T19:44:17.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.265+0000 7f65d97fa640 1 -- 172.21.15.29:0/2683293821 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d40039e0 con 0x7f65e40ba0e0 2024-04-25T19:44:17.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.265+0000 7f65eaaf1640 1 -- 172.21.15.29:0/2683293821 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65e413c410 con 0x7f65e40ba0e0 2024-04-25T19:44:17.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.265+0000 7f65eaaf1640 1 -- 172.21.15.29:0/2683293821 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f65e40b5220 con 0x7f65e40ba0e0 2024-04-25T19:44:17.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.267+0000 7f65eaaf1640 1 -- 172.21.15.29:0/2683293821 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65e40507b0 con 0x7f65e40ba0e0 2024-04-25T19:44:17.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.268+0000 7f65d97fa640 1 -- 172.21.15.29:0/2683293821 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f65d401a070 con 0x7f65e40ba0e0 2024-04-25T19:44:17.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.268+0000 7f65d97fa640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:17.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.269+0000 7f65d97fa640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:17.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.269+0000 7f65d97fa640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:17.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.269+0000 7f65d97fa640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:17.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.269+0000 7f65d97fa640 1 --2- 172.21.15.29:0/2683293821 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f65d00ac530 0x7f65d00aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.269+0000 7f65d97fa640 1 -- 172.21.15.29:0/2683293821 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f65d40d86a0 con 0x7f65e40ba0e0 2024-04-25T19:44:17.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.269+0000 7f65e888d640 1 --2- 172.21.15.29:0/2683293821 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f65d00ac530 0x7f65d00aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.270+0000 7f65e888d640 1 --2- 172.21.15.29:0/2683293821 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f65d00ac530 0x7f65d00aea40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009560 tx=0x7f65e0009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:17.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.278+0000 7f65d97fa640 1 -- 172.21.15.29:0/2683293821 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f65d404eb80 con 0x7f65e40ba0e0 2024-04-25T19:44:17.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.297+0000 7fec902c2640 1 Processor -- start 2024-04-25T19:44:17.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.297+0000 7fec902c2640 1 -- start start 2024-04-25T19:44:17.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.299+0000 7fec902c2640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fec88072a90 0x7fec8806f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.299+0000 7fec902c2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec8806cb90 con 0x7fec88072680 2024-04-25T19:44:17.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.299+0000 7fec902c2640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec880734c0 con 0x7fec8806fb60 2024-04-25T19:44:17.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.299+0000 7fec902c2640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fec8806ff70 con 0x7fec88072a90 2024-04-25T19:44:17.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.299+0000 7fec8d85d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fec88072a90 0x7fec8806f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.299+0000 7fec8d85d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fec88072a90 0x7fec8806f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34622/0 (socket says 172.21.15.29:34622) 2024-04-25T19:44:17.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.299+0000 7fec8d85d640 1 -- 172.21.15.29:0/3595015380 learned_addr learned my addr 172.21.15.29:0/3595015380 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:17.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.299+0000 7fec8e05e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fec88072680 0x7fec880a6d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38080/0 (socket says 172.21.15.29:38080) 2024-04-25T19:44:17.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.299+0000 7fec8d85d640 1 -- 172.21.15.29:0/3595015380 >> v1:172.21.15.102:6789/0 conn(0x7fec8806fb60 legacy=0x7fec88073780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.299+0000 7fec8d85d640 1 -- 172.21.15.29:0/3595015380 >> v1:172.21.15.29:6789/0 conn(0x7fec88072680 legacy=0x7fec880a6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.299+0000 7fec8d85d640 1 -- 172.21.15.29:0/3595015380 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec880a7490 con 0x7fec88072a90 2024-04-25T19:44:17.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.299+0000 7fec8d05c640 1 -- 172.21.15.29:0/3595015380 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2463403569 0 0) 0x7fec8806cb90 con 0x7fec88072680 2024-04-25T19:44:17.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.300+0000 7fec8d85d640 1 --2- 172.21.15.29:0/3595015380 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fec88072a90 0x7fec8806f620 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fec780098a0 tx=0x7fec7802ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9bc1534cf3ab9f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:17.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.300+0000 7fec8d05c640 1 -- 172.21.15.29:0/3595015380 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7802f830 con 0x7fec88072a90 2024-04-25T19:44:17.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.300+0000 7fec8d05c640 1 -- 172.21.15.29:0/3595015380 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec7802f9d0 con 0x7fec88072a90 2024-04-25T19:44:17.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.300+0000 7fec8d05c640 1 -- 172.21.15.29:0/3595015380 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7802fda0 con 0x7fec88072a90 2024-04-25T19:44:17.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.301+0000 7fec902c2640 1 -- 172.21.15.29:0/3595015380 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fec88072a90 msgr2=0x7fec8806f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.301+0000 7fec902c2640 1 --2- 172.21.15.29:0/3595015380 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fec88072a90 0x7fec8806f620 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fec780098a0 tx=0x7fec7802ba10 comp rx=0 tx=0).stop 2024-04-25T19:44:17.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.301+0000 7fec902c2640 1 -- 172.21.15.29:0/3595015380 shutdown_connections 2024-04-25T19:44:17.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.301+0000 7fec902c2640 1 --2- 172.21.15.29:0/3595015380 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fec88072a90 0x7fec8806f620 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.301+0000 7fec902c2640 1 -- 172.21.15.29:0/3595015380 >> 172.21.15.29:0/3595015380 conn(0x7fec880a7d40 msgr2=0x7fec880a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:17.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.301+0000 7fec902c2640 1 -- 172.21.15.29:0/3595015380 shutdown_connections 2024-04-25T19:44:17.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.301+0000 7fec902c2640 1 -- 172.21.15.29:0/3595015380 wait complete. 2024-04-25T19:44:17.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.303+0000 7fec902c2640 1 Processor -- start 2024-04-25T19:44:17.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.303+0000 7fec902c2640 1 -- start start 2024-04-25T19:44:17.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.304+0000 7fec902c2640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fec8806fb60 0x7fec881cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.304+0000 7fec902c2640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec88072680 0x7fec88137c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.304+0000 7fec902c2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec88072a90 0x7fec88138c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.304+0000 7fec902c2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fec88074010 con 0x7fec88072a90 2024-04-25T19:44:17.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.304+0000 7fec902c2640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fec88073e90 con 0x7fec88072680 2024-04-25T19:44:17.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.304+0000 7fec902c2640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fec88074190 con 0x7fec8806fb60 2024-04-25T19:44:17.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.304+0000 7fec8e05e640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fec8806fb60 0x7fec881cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.304+0000 7fec8e05e640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fec8806fb60 0x7fec881cbeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34630/0 (socket says 172.21.15.29:34630) 2024-04-25T19:44:17.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.304+0000 7fec8e05e640 1 -- 172.21.15.29:0/979590215 learned_addr learned my addr 172.21.15.29:0/979590215 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:17.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.304+0000 7fec8d85d640 1 --2- 172.21.15.29:0/979590215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec88072680 0x7fec88137c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.304+0000 7fec8e85f640 1 --2- 172.21.15.29:0/979590215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec88072a90 0x7fec88138c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.305+0000 7fec8d85d640 1 -- 172.21.15.29:0/979590215 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fec8806fb60 msgr2=0x7fec881cbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.305+0000 7fec8d85d640 1 --2- 172.21.15.29:0/979590215 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fec8806fb60 0x7fec881cbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.305+0000 7fec8d85d640 1 -- 172.21.15.29:0/979590215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec88072a90 msgr2=0x7fec88138c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.305+0000 7fec8d85d640 1 --2- 172.21.15.29:0/979590215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec88072a90 0x7fec88138c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.305+0000 7fec8d85d640 1 -- 172.21.15.29:0/979590215 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec881381d0 con 0x7fec88072680 2024-04-25T19:44:17.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.305+0000 7fec8e85f640 1 --2- 172.21.15.29:0/979590215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec88072a90 0x7fec88138c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:44:17.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.305+0000 7fec8d85d640 1 --2- 172.21.15.29:0/979590215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec88072680 0x7fec88137c90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fec8806d030 tx=0x7fec78004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:17.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.305+0000 7fec7effd640 1 -- 172.21.15.29:0/979590215 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec78047070 con 0x7fec88072680 2024-04-25T19:44:17.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.305+0000 7fec7effd640 1 -- 172.21.15.29:0/979590215 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec78004030 con 0x7fec88072680 2024-04-25T19:44:17.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.305+0000 7fec7effd640 1 -- 172.21.15.29:0/979590215 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec78004420 con 0x7fec88072680 2024-04-25T19:44:17.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.305+0000 7fec902c2640 1 -- 172.21.15.29:0/979590215 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec8813a060 con 0x7fec88072680 2024-04-25T19:44:17.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.305+0000 7fec902c2640 1 -- 172.21.15.29:0/979590215 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec881dc4e0 con 0x7fec88072680 2024-04-25T19:44:17.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.306+0000 7fec902c2640 1 -- 172.21.15.29:0/979590215 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec880507b0 con 0x7fec88072680 2024-04-25T19:44:17.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.308+0000 7fec7effd640 1 -- 172.21.15.29:0/979590215 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7fec78003610 con 0x7fec88072680 2024-04-25T19:44:17.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.308+0000 7fec7effd640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:17.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.308+0000 7fec7effd640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:17.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.308+0000 7fec7effd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:17.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.308+0000 7fec7effd640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:17.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.308+0000 7fec7effd640 1 --2- 172.21.15.29:0/979590215 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fec6c0ac4b0 0x7fec6c0ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.309+0000 7fec7effd640 1 -- 172.21.15.29:0/979590215 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7fec780fb7a0 con 0x7fec88072680 2024-04-25T19:44:17.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.309+0000 7fec8e05e640 1 --2- 172.21.15.29:0/979590215 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fec6c0ac4b0 0x7fec6c0ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.309+0000 7fec8e05e640 1 --2- 172.21.15.29:0/979590215 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fec6c0ac4b0 0x7fec6c0ae9c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fec84000f60 tx=0x7fec84007240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:17.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.312+0000 7fec7effd640 1 -- 172.21.15.29:0/979590215 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fec78050080 con 0x7fec88072680 2024-04-25T19:44:17.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.348+0000 7f673db1b640 1 Processor -- start 2024-04-25T19:44:17.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.348+0000 7f673db1b640 1 -- start start 2024-04-25T19:44:17.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.349+0000 7f673db1b640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f673806da90 0x7f67380a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.349+0000 7f673db1b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f67381472d0 0x7f67381476e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.349+0000 7f673db1b640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6738146c60 con 0x7f67380acd40 2024-04-25T19:44:17.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.349+0000 7f673db1b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f67380a5c10 con 0x7f67381472d0 2024-04-25T19:44:17.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.349+0000 7f673db1b640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f67380ad9d0 con 0x7f673806da90 2024-04-25T19:44:17.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.349+0000 7f672effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f67381472d0 0x7f67381476e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.349+0000 7f672effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f67381472d0 0x7f67381476e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36650/0 (socket says 172.21.15.29:36650) 2024-04-25T19:44:17.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.349+0000 7f672effd640 1 -- 172.21.15.29:0/730050158 learned_addr learned my addr 172.21.15.29:0/730050158 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:17.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.349+0000 7f67377fe640 1 --2- 172.21.15.29:0/730050158 >> v2:172.21.15.102:3300/0 conn(0x7f673806da90 0x7f67380a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.349+0000 7f672effd640 1 -- 172.21.15.29:0/730050158 >> v2:172.21.15.102:3300/0 conn(0x7f673806da90 msgr2=0x7f67380a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:17.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.349+0000 7f672effd640 1 --2- 172.21.15.29:0/730050158 >> v2:172.21.15.102:3300/0 conn(0x7f673806da90 0x7f67380a56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.349+0000 7f672effd640 1 -- 172.21.15.29:0/730050158 >> v1:172.21.15.102:6789/0 conn(0x7f67380acd40 legacy=0x7f67380ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.349+0000 7f672effd640 1 -- 172.21.15.29:0/730050158 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6738152400 con 0x7f67381472d0 2024-04-25T19:44:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.350+0000 7f672effd640 1 --2- 172.21.15.29:0/730050158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67381472d0 0x7f67381476e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f673806f110 tx=0x7f672002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4185279513bff9e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.350+0000 7f6736ffd640 1 -- 172.21.15.29:0/730050158 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720034070 con 0x7f67381472d0 2024-04-25T19:44:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.350+0000 7f6736ffd640 1 -- 172.21.15.29:0/730050158 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6720030860 con 0x7f67381472d0 2024-04-25T19:44:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.350+0000 7f6736ffd640 1 -- 172.21.15.29:0/730050158 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720030c30 con 0x7f67381472d0 2024-04-25T19:44:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.351+0000 7f673db1b640 1 -- 172.21.15.29:0/730050158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67381472d0 msgr2=0x7f67381476e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.351+0000 7f673db1b640 1 --2- 172.21.15.29:0/730050158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67381472d0 0x7f67381476e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f673806f110 tx=0x7f672002b9a0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.351+0000 7f673db1b640 1 -- 172.21.15.29:0/730050158 shutdown_connections 2024-04-25T19:44:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.351+0000 7f673db1b640 1 --2- 172.21.15.29:0/730050158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67381472d0 0x7f67381476e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.351+0000 7f673db1b640 1 --2- 172.21.15.29:0/730050158 >> v2:172.21.15.102:3300/0 conn(0x7f673806da90 0x7f67380a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.351+0000 7f673db1b640 1 -- 172.21.15.29:0/730050158 >> 172.21.15.29:0/730050158 conn(0x7f67380bc3b0 msgr2=0x7f67380bc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.351+0000 7f673db1b640 1 -- 172.21.15.29:0/730050158 shutdown_connections 2024-04-25T19:44:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.351+0000 7f673db1b640 1 -- 172.21.15.29:0/730050158 wait complete. 2024-04-25T19:44:17.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.353+0000 7f673db1b640 1 Processor -- start 2024-04-25T19:44:17.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.353+0000 7f673db1b640 1 -- start start 2024-04-25T19:44:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.354+0000 7f673db1b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f673806da90 0x7f6738141b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.354+0000 7f673db1b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f67380acd40 0x7f673813b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.354+0000 7f673db1b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67381472d0 0x7f673813bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.354+0000 7f673db1b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6738151780 con 0x7f673806da90 2024-04-25T19:44:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.354+0000 7f673db1b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6738151600 con 0x7f67381472d0 2024-04-25T19:44:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.354+0000 7f673db1b640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f6738151900 con 0x7f67380acd40 2024-04-25T19:44:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.354+0000 7f67377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f673806da90 0x7f6738141b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.354+0000 7f672effd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f67380acd40 0x7f673813b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.354+0000 7f67377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f673806da90 0x7f6738141b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36664/0 (socket says 172.21.15.29:36664) 2024-04-25T19:44:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.354+0000 7f67377fe640 1 -- 172.21.15.29:0/2835737098 learned_addr learned my addr 172.21.15.29:0/2835737098 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.354+0000 7f67377fe640 1 -- 172.21.15.29:0/2835737098 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f67380acd40 msgr2=0x7f673813b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.354+0000 7f67377fe640 1 --2- 172.21.15.29:0/2835737098 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f67380acd40 0x7f673813b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.354+0000 7f67377fe640 1 -- 172.21.15.29:0/2835737098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67381472d0 msgr2=0x7f673813bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.354+0000 7f67377fe640 1 --2- 172.21.15.29:0/2835737098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67381472d0 0x7f673813bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.354+0000 7f67377fe640 1 -- 172.21.15.29:0/2835737098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f673813d2d0 con 0x7f673806da90 2024-04-25T19:44:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.355+0000 7f67377fe640 1 --2- 172.21.15.29:0/2835737098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f673806da90 0x7f6738141b90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f672800e8a0 tx=0x7f672800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.355+0000 7f6734ff9640 1 -- 172.21.15.29:0/2835737098 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6728017070 con 0x7f673806da90 2024-04-25T19:44:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.355+0000 7f6734ff9640 1 -- 172.21.15.29:0/2835737098 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f672800c940 con 0x7f673806da90 2024-04-25T19:44:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.355+0000 7f673db1b640 1 -- 172.21.15.29:0/2835737098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67380b6970 con 0x7f673806da90 2024-04-25T19:44:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.355+0000 7f6734ff9640 1 -- 172.21.15.29:0/2835737098 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672800cd30 con 0x7f673806da90 2024-04-25T19:44:17.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.355+0000 7f673db1b640 1 -- 172.21.15.29:0/2835737098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f673813f2e0 con 0x7f673806da90 2024-04-25T19:44:17.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.356+0000 7f673db1b640 1 -- 172.21.15.29:0/2835737098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f673806e5a0 con 0x7f673806da90 2024-04-25T19:44:17.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.357+0000 7f6734ff9640 1 -- 172.21.15.29:0/2835737098 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f672801a070 con 0x7f673806da90 2024-04-25T19:44:17.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.357+0000 7f6734ff9640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:17.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.358+0000 7f6734ff9640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:17.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.358+0000 7f6734ff9640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:17.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.358+0000 7f6734ff9640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:17.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.358+0000 7f6734ff9640 1 --2- 172.21.15.29:0/2835737098 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f67180ac530 0x7f67180aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.358+0000 7f6734ff9640 1 -- 172.21.15.29:0/2835737098 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f67280d9310 con 0x7f673806da90 2024-04-25T19:44:17.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.358+0000 7f672effd640 1 --2- 172.21.15.29:0/2835737098 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f67180ac530 0x7f67180aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.359+0000 7f672effd640 1 --2- 172.21.15.29:0/2835737098 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f67180ac530 0x7f67180aea40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6720036000 tx=0x7f6720008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:17.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.366+0000 7f6734ff9640 1 -- 172.21.15.29:0/2835737098 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f673806e5a0 con 0x7f673806da90 2024-04-25T19:44:17.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.373+0000 7f3076bc4640 1 Processor -- start 2024-04-25T19:44:17.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.373+0000 7f3076bc4640 1 -- start start 2024-04-25T19:44:17.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.374+0000 7f3076bc4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f307014d2f0 0x7f307014f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.374+0000 7f3076bc4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30700a9340 con 0x7f30700acce0 2024-04-25T19:44:17.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.374+0000 7f3075161640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f307014d2f0 0x7f307014f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.374+0000 7f3076bc4640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30700aa490 con 0x7f30700a9800 2024-04-25T19:44:17.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.374+0000 7f3076bc4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f30700aa750 con 0x7f307014d2f0 2024-04-25T19:44:17.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.374+0000 7f3075161640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f307014d2f0 0x7f307014f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36670/0 (socket says 172.21.15.29:36670) 2024-04-25T19:44:17.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.374+0000 7f3075161640 1 -- 172.21.15.29:0/1769502891 learned_addr learned my addr 172.21.15.29:0/1769502891 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:17.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.375+0000 7f3075161640 1 -- 172.21.15.29:0/1769502891 >> v1:172.21.15.102:6789/0 conn(0x7f30700a9800 legacy=0x7f30700a9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.375+0000 7f3075161640 1 -- 172.21.15.29:0/1769502891 >> v1:172.21.15.29:6789/0 conn(0x7f30700acce0 legacy=0x7f30700ad0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.375+0000 7f3075161640 1 -- 172.21.15.29:0/1769502891 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3070072800 con 0x7f307014d2f0 2024-04-25T19:44:17.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.376+0000 7f3075161640 1 --2- 172.21.15.29:0/1769502891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f307014d2f0 0x7f307014f710 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f306800d150 tx=0x7f306802e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5921377b42ff9e39 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:17.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.376+0000 7f306f7fe640 1 -- 172.21.15.29:0/1769502891 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306802e960 con 0x7f307014d2f0 2024-04-25T19:44:17.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.376+0000 7f306f7fe640 1 -- 172.21.15.29:0/1769502891 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f306802eb00 con 0x7f307014d2f0 2024-04-25T19:44:17.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.376+0000 7f306f7fe640 1 -- 172.21.15.29:0/1769502891 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3068009600 con 0x7f307014d2f0 2024-04-25T19:44:17.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.377+0000 7f3076bc4640 1 -- 172.21.15.29:0/1769502891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f307014d2f0 msgr2=0x7f307014f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.377+0000 7f3076bc4640 1 --2- 172.21.15.29:0/1769502891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f307014d2f0 0x7f307014f710 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f306800d150 tx=0x7f306802e040 comp rx=0 tx=0).stop 2024-04-25T19:44:17.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.377+0000 7f3076bc4640 1 -- 172.21.15.29:0/1769502891 shutdown_connections 2024-04-25T19:44:17.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.377+0000 7f3076bc4640 1 --2- 172.21.15.29:0/1769502891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f307014d2f0 0x7f307014f710 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.377+0000 7f3076bc4640 1 -- 172.21.15.29:0/1769502891 >> 172.21.15.29:0/1769502891 conn(0x7f307006d6a0 msgr2=0x7f307006dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:17.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.377+0000 7f3076bc4640 1 -- 172.21.15.29:0/1769502891 shutdown_connections 2024-04-25T19:44:17.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.377+0000 7f3076bc4640 1 -- 172.21.15.29:0/1769502891 wait complete. 2024-04-25T19:44:17.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.378+0000 7f3076bc4640 1 Processor -- start 2024-04-25T19:44:17.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.379+0000 7f3076bc4640 1 -- start start 2024-04-25T19:44:17.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.380+0000 7f3076bc4640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f30700a9800 0x7f30700a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.380+0000 7f3076bc4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30700acce0 0x7f30700a5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.380+0000 7f3076bc4640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f307014d2f0 0x7f30701dc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.380+0000 7f3076bc4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f307013a280 con 0x7f30700acce0 2024-04-25T19:44:17.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.380+0000 7f3076bc4640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f307013a100 con 0x7f307014d2f0 2024-04-25T19:44:17.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.380+0000 7f3076bc4640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f307006c590 con 0x7f30700a9800 2024-04-25T19:44:17.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.380+0000 7f3074960640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f30700a9800 0x7f30700a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.380+0000 7f3074960640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f30700a9800 0x7f30700a30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34654/0 (socket says 172.21.15.29:34654) 2024-04-25T19:44:17.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.380+0000 7f3074960640 1 -- 172.21.15.29:0/3843696173 learned_addr learned my addr 172.21.15.29:0/3843696173 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:17.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.380+0000 7f306ffff640 1 --2- 172.21.15.29:0/3843696173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30700acce0 0x7f30700a5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.380+0000 7f3075161640 1 --2- 172.21.15.29:0/3843696173 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f307014d2f0 0x7f30701dc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.380+0000 7f3074960640 1 -- 172.21.15.29:0/3843696173 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f307014d2f0 msgr2=0x7f30701dc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.380+0000 7f3074960640 1 --2- 172.21.15.29:0/3843696173 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f307014d2f0 0x7f30701dc620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.380+0000 7f3074960640 1 -- 172.21.15.29:0/3843696173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30700acce0 msgr2=0x7f30700a5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.380+0000 7f3074960640 1 --2- 172.21.15.29:0/3843696173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30700acce0 0x7f30700a5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.380+0000 7f3074960640 1 -- 172.21.15.29:0/3843696173 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30700a6320 con 0x7f30700a9800 2024-04-25T19:44:17.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.381+0000 7f3075161640 1 --2- 172.21.15.29:0/3843696173 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f307014d2f0 0x7f30701dc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:44:17.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.381+0000 7f306ffff640 1 --2- 172.21.15.29:0/3843696173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30700acce0 0x7f30700a5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:17.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.381+0000 7f3074960640 1 --2- 172.21.15.29:0/3843696173 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f30700a9800 0x7f30700a30b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f30640019b0 tx=0x7f3064001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:17.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.381+0000 7f306d7fa640 1 -- 172.21.15.29:0/3843696173 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3064010300 con 0x7f30700a9800 2024-04-25T19:44:17.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.381+0000 7f306d7fa640 1 -- 172.21.15.29:0/3843696173 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30640050d0 con 0x7f30700a9800 2024-04-25T19:44:17.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.381+0000 7f3076bc4640 1 -- 172.21.15.29:0/3843696173 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3070141e60 con 0x7f30700a9800 2024-04-25T19:44:17.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.381+0000 7f306d7fa640 1 -- 172.21.15.29:0/3843696173 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306400a700 con 0x7f30700a9800 2024-04-25T19:44:17.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.381+0000 7f3076bc4640 1 -- 172.21.15.29:0/3843696173 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f30701dcd40 con 0x7f30700a9800 2024-04-25T19:44:17.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.382+0000 7f3076bc4640 1 -- 172.21.15.29:0/3843696173 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30701dd300 con 0x7f30700a9800 2024-04-25T19:44:17.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.384+0000 7f306d7fa640 1 -- 172.21.15.29:0/3843696173 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f306400a8a0 con 0x7f30700a9800 2024-04-25T19:44:17.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.384+0000 7f306d7fa640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:17.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.384+0000 7f306d7fa640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:17.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.384+0000 7f306d7fa640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:17.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.384+0000 7f306d7fa640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:17.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.384+0000 7f306d7fa640 1 --2- 172.21.15.29:0/3843696173 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f30540ac560 0x7f30540aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.385+0000 7f306d7fa640 1 -- 172.21.15.29:0/3843696173 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f30640dd170 con 0x7f30700a9800 2024-04-25T19:44:17.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.385+0000 7f306ffff640 1 --2- 172.21.15.29:0/3843696173 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f30540ac560 0x7f30540aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.385+0000 7f306ffff640 1 --2- 172.21.15.29:0/3843696173 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f30540ac560 0x7f30540aea70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3070144f00 tx=0x7f305c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:17.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.393+0000 7f306d7fa640 1 -- 172.21.15.29:0/3843696173 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f30640a7de0 con 0x7f30700a9800 2024-04-25T19:44:17.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.611+0000 7f65eaaf1640 1 -- 172.21.15.29:0/2683293821 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f65e406db00 con 0x7f65e40ba0e0 2024-04-25T19:44:17.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.612+0000 7f65d97fa640 1 -- 172.21.15.29:0/2683293821 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f65d40a2ca0 con 0x7f65e40ba0e0 2024-04-25T19:44:17.614 INFO:teuthology.orchestra.run.smithi029.stdout:38654705670 2024-04-25T19:44:17.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.615+0000 7f65eaaf1640 10 mgrc shutdown 2024-04-25T19:44:17.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.615+0000 7f65eaaf1640 1 -- 172.21.15.29:0/2683293821 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f65d00ac530 msgr2=0x7f65d00aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.615+0000 7f65eaaf1640 1 --2- 172.21.15.29:0/2683293821 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f65d00ac530 0x7f65d00aea40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009560 tx=0x7f65e0009190 comp rx=0 tx=0).stop 2024-04-25T19:44:17.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.615+0000 7f65eaaf1640 1 -- 172.21.15.29:0/2683293821 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f65e40ba0e0 msgr2=0x7f65e40b4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.615+0000 7f65eaaf1640 1 --2- 172.21.15.29:0/2683293821 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f65e40ba0e0 0x7f65e40b4b00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f65d400c3c0 tx=0x7f65d400c880 comp rx=0 tx=0).stop 2024-04-25T19:44:17.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.615+0000 7f65eaaf1640 1 -- 172.21.15.29:0/2683293821 shutdown_connections 2024-04-25T19:44:17.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.615+0000 7f65eaaf1640 1 --2- 172.21.15.29:0/2683293821 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f65d00ac530 0x7f65d00aea40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.615+0000 7f65eaaf1640 1 --2- 172.21.15.29:0/2683293821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65e41442b0 0x7f65e4138270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.615+0000 7f65eaaf1640 1 --2- 172.21.15.29:0/2683293821 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f65e40ba0e0 0x7f65e40b4b00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.615+0000 7f65eaaf1640 1 --2- 172.21.15.29:0/2683293821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65e40aba00 0x7f65e41404d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.615+0000 7f65eaaf1640 1 -- 172.21.15.29:0/2683293821 >> 172.21.15.29:0/2683293821 conn(0x7f65e4072ab0 msgr2=0x7f65e4145df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:17.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.615+0000 7f65eaaf1640 1 -- 172.21.15.29:0/2683293821 shutdown_connections 2024-04-25T19:44:17.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.615+0000 7f65eaaf1640 1 -- 172.21.15.29:0/2683293821 wait complete. 2024-04-25T19:44:17.628 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.0 2024-04-25T19:44:17.628 DEBUG:teuthology.parallel:result is None 2024-04-25T19:44:17.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.640+0000 7fec902c2640 1 -- 172.21.15.29:0/979590215 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fec88070620 con 0x7fec88072680 2024-04-25T19:44:17.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.641+0000 7fec7effd640 1 -- 172.21.15.29:0/979590215 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fec78033030 con 0x7fec88072680 2024-04-25T19:44:17.642 INFO:teuthology.orchestra.run.smithi029.stdout:42949672967 2024-04-25T19:44:17.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.643+0000 7fec902c2640 10 mgrc shutdown 2024-04-25T19:44:17.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.643+0000 7fec902c2640 1 -- 172.21.15.29:0/979590215 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fec6c0ac4b0 msgr2=0x7fec6c0ae9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.643+0000 7fec902c2640 1 --2- 172.21.15.29:0/979590215 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fec6c0ac4b0 0x7fec6c0ae9c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fec84000f60 tx=0x7fec84007240 comp rx=0 tx=0).stop 2024-04-25T19:44:17.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.643+0000 7fec902c2640 1 -- 172.21.15.29:0/979590215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec88072680 msgr2=0x7fec88137c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.643+0000 7fec902c2640 1 --2- 172.21.15.29:0/979590215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec88072680 0x7fec88137c90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fec8806d030 tx=0x7fec78004730 comp rx=0 tx=0).stop 2024-04-25T19:44:17.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.643+0000 7fec902c2640 1 -- 172.21.15.29:0/979590215 shutdown_connections 2024-04-25T19:44:17.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.643+0000 7fec902c2640 1 --2- 172.21.15.29:0/979590215 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fec6c0ac4b0 0x7fec6c0ae9c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.643+0000 7fec902c2640 1 --2- 172.21.15.29:0/979590215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec88072a90 0x7fec88138c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.643+0000 7fec902c2640 1 --2- 172.21.15.29:0/979590215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec88072680 0x7fec88137c90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.643+0000 7fec902c2640 1 --2- 172.21.15.29:0/979590215 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fec8806fb60 0x7fec881cbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.643+0000 7fec902c2640 1 -- 172.21.15.29:0/979590215 >> 172.21.15.29:0/979590215 conn(0x7fec880a7d40 msgr2=0x7fec881d1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:17.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.643+0000 7fec902c2640 1 -- 172.21.15.29:0/979590215 shutdown_connections 2024-04-25T19:44:17.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.643+0000 7fec902c2640 1 -- 172.21.15.29:0/979590215 wait complete. 2024-04-25T19:44:17.655 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672967 got 42949672967 for osd.3 2024-04-25T19:44:17.655 DEBUG:teuthology.parallel:result is None 2024-04-25T19:44:17.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.677+0000 7f3076bc4640 1 -- 172.21.15.29:0/3843696173 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f30701dd020 con 0x7f30700a9800 2024-04-25T19:44:17.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.677+0000 7f306d7fa640 1 -- 172.21.15.29:0/3843696173 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f30640a76e0 con 0x7f30700a9800 2024-04-25T19:44:17.679 INFO:teuthology.orchestra.run.smithi029.stdout:42949672967 2024-04-25T19:44:17.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.680+0000 7f673db1b640 1 -- 172.21.15.29:0/2835737098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f67380507b0 con 0x7f673806da90 2024-04-25T19:44:17.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.680+0000 7f3076bc4640 10 mgrc shutdown 2024-04-25T19:44:17.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.680+0000 7f3076bc4640 1 -- 172.21.15.29:0/3843696173 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f30540ac560 msgr2=0x7f30540aea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.680+0000 7f3076bc4640 1 --2- 172.21.15.29:0/3843696173 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f30540ac560 0x7f30540aea70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3070144f00 tx=0x7f305c009040 comp rx=0 tx=0).stop 2024-04-25T19:44:17.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.680+0000 7f3076bc4640 1 -- 172.21.15.29:0/3843696173 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f30700a9800 msgr2=0x7f30700a30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.680+0000 7f3076bc4640 1 --2- 172.21.15.29:0/3843696173 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f30700a9800 0x7f30700a30b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f30640019b0 tx=0x7f3064001e70 comp rx=0 tx=0).stop 2024-04-25T19:44:17.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.680+0000 7f3076bc4640 1 -- 172.21.15.29:0/3843696173 shutdown_connections 2024-04-25T19:44:17.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.680+0000 7f3076bc4640 1 --2- 172.21.15.29:0/3843696173 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f30540ac560 0x7f30540aea70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.680+0000 7f3076bc4640 1 --2- 172.21.15.29:0/3843696173 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f307014d2f0 0x7f30701dc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.680+0000 7f3076bc4640 1 --2- 172.21.15.29:0/3843696173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30700acce0 0x7f30700a5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.680+0000 7f3076bc4640 1 --2- 172.21.15.29:0/3843696173 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f30700a9800 0x7f30700a30b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.680+0000 7f3076bc4640 1 -- 172.21.15.29:0/3843696173 >> 172.21.15.29:0/3843696173 conn(0x7f307006d6a0 msgr2=0x7f3070136530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:17.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.680+0000 7f3076bc4640 1 -- 172.21.15.29:0/3843696173 shutdown_connections 2024-04-25T19:44:17.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.680+0000 7f3076bc4640 1 -- 172.21.15.29:0/3843696173 wait complete. 2024-04-25T19:44:17.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.680+0000 7f6734ff9640 1 -- 172.21.15.29:0/2835737098 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f67280a3ee0 con 0x7f673806da90 2024-04-25T19:44:17.683 INFO:teuthology.orchestra.run.smithi029.stdout:42949672967 2024-04-25T19:44:17.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.682+0000 7f673db1b640 10 mgrc shutdown 2024-04-25T19:44:17.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.683+0000 7f673db1b640 1 -- 172.21.15.29:0/2835737098 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f67180ac530 msgr2=0x7f67180aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.683+0000 7f673db1b640 1 --2- 172.21.15.29:0/2835737098 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f67180ac530 0x7f67180aea40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6720036000 tx=0x7f6720008030 comp rx=0 tx=0).stop 2024-04-25T19:44:17.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.683+0000 7f673db1b640 1 -- 172.21.15.29:0/2835737098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f673806da90 msgr2=0x7f6738141b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.683+0000 7f673db1b640 1 --2- 172.21.15.29:0/2835737098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f673806da90 0x7f6738141b90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f672800e8a0 tx=0x7f672800ed60 comp rx=0 tx=0).stop 2024-04-25T19:44:17.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.683+0000 7f673db1b640 1 -- 172.21.15.29:0/2835737098 shutdown_connections 2024-04-25T19:44:17.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.683+0000 7f673db1b640 1 --2- 172.21.15.29:0/2835737098 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f67180ac530 0x7f67180aea40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.683+0000 7f673db1b640 1 --2- 172.21.15.29:0/2835737098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67381472d0 0x7f673813bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.683+0000 7f673db1b640 1 --2- 172.21.15.29:0/2835737098 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f67380acd40 0x7f673813b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.683+0000 7f673db1b640 1 --2- 172.21.15.29:0/2835737098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f673806da90 0x7f6738141b90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.683+0000 7f673db1b640 1 -- 172.21.15.29:0/2835737098 >> 172.21.15.29:0/2835737098 conn(0x7f67380bc3b0 msgr2=0x7f67380a9630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:17.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.683+0000 7f673db1b640 1 -- 172.21.15.29:0/2835737098 shutdown_connections 2024-04-25T19:44:17.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.683+0000 7f673db1b640 1 -- 172.21.15.29:0/2835737098 wait complete. 2024-04-25T19:44:17.693 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672967 got 42949672967 for osd.1 2024-04-25T19:44:17.693 DEBUG:teuthology.parallel:result is None 2024-04-25T19:44:17.695 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672967 got 42949672967 for osd.2 2024-04-25T19:44:17.695 DEBUG:teuthology.parallel:result is None 2024-04-25T19:44:17.695 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T19:44:17.695 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T19:44:17.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.846+0000 7f5deb954640 1 Processor -- start 2024-04-25T19:44:17.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.846+0000 7f5deb954640 1 -- start start 2024-04-25T19:44:17.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.847+0000 7f5deb954640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5de406df40 0x7f5de40aca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.847+0000 7f5deb954640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5de40acf70 0x7f5de4151400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.847+0000 7f5deb954640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5de40ad500 con 0x7f5de4146640 2024-04-25T19:44:17.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.847+0000 7f5deb954640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f5de40ad7c0 con 0x7f5de40acf70 2024-04-25T19:44:17.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.847+0000 7f5deb954640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f5de40ae650 con 0x7f5de406df40 2024-04-25T19:44:17.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.847+0000 7f5de8eef640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5de406df40 0x7f5de40aca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.847+0000 7f5de8eef640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5de406df40 0x7f5de40aca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:35084/0 (socket says 172.21.15.29:35084) 2024-04-25T19:44:17.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.847+0000 7f5de8eef640 1 -- 172.21.15.29:0/3837270624 learned_addr learned my addr 172.21.15.29:0/3837270624 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:17.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.847+0000 7f5de9ef1640 1 --2- 172.21.15.29:0/3837270624 >> v2:172.21.15.29:3300/0 conn(0x7f5de40acf70 0x7f5de4151400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.847+0000 7f5de9ef1640 1 -- 172.21.15.29:0/3837270624 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5de406df40 msgr2=0x7f5de40aca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:17.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.847+0000 7f5de9ef1640 1 --2- 172.21.15.29:0/3837270624 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5de406df40 0x7f5de40aca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.847+0000 7f5de9ef1640 1 -- 172.21.15.29:0/3837270624 >> v1:172.21.15.102:6789/0 conn(0x7f5de4146640 legacy=0x7f5de40aa840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.847+0000 7f5de9ef1640 1 -- 172.21.15.29:0/3837270624 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de406e350 con 0x7f5de40acf70 2024-04-25T19:44:17.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.847+0000 7f5de8eef640 1 --2- 172.21.15.29:0/3837270624 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5de406df40 0x7f5de40aca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:17.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.848+0000 7f5de9ef1640 1 --2- 172.21.15.29:0/3837270624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5de40acf70 0x7f5de4151400 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4009d90 tx=0x7f5dd402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03a162caa25f051 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:17.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.848+0000 7f5de3fff640 1 -- 172.21.15.29:0/3837270624 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4039070 con 0x7f5de40acf70 2024-04-25T19:44:17.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.848+0000 7f5de3fff640 1 -- 172.21.15.29:0/3837270624 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dd402cce0 con 0x7f5de40acf70 2024-04-25T19:44:17.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.848+0000 7f5de3fff640 1 -- 172.21.15.29:0/3837270624 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4034660 con 0x7f5de40acf70 2024-04-25T19:44:17.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.849+0000 7f5deb954640 1 -- 172.21.15.29:0/3837270624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5de40acf70 msgr2=0x7f5de4151400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.849+0000 7f5deb954640 1 --2- 172.21.15.29:0/3837270624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5de40acf70 0x7f5de4151400 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4009d90 tx=0x7f5dd402bf30 comp rx=0 tx=0).stop 2024-04-25T19:44:17.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.849+0000 7f5deb954640 1 -- 172.21.15.29:0/3837270624 shutdown_connections 2024-04-25T19:44:17.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.849+0000 7f5deb954640 1 --2- 172.21.15.29:0/3837270624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5de40acf70 0x7f5de4151400 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.849+0000 7f5deb954640 1 --2- 172.21.15.29:0/3837270624 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5de406df40 0x7f5de40aca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.849+0000 7f5deb954640 1 -- 172.21.15.29:0/3837270624 >> 172.21.15.29:0/3837270624 conn(0x7f5de40ab710 msgr2=0x7f5de40bbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:17.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.849+0000 7f5deb954640 1 -- 172.21.15.29:0/3837270624 shutdown_connections 2024-04-25T19:44:17.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.849+0000 7f5deb954640 1 -- 172.21.15.29:0/3837270624 wait complete. 2024-04-25T19:44:17.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.850+0000 7f5deb954640 1 Processor -- start 2024-04-25T19:44:17.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.850+0000 7f5deb954640 1 -- start start 2024-04-25T19:44:17.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.851+0000 7f5deb954640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5de406df40 0x7f5de41453c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.851+0000 7f5deb954640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5de40acf70 0x7f5de413fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.851+0000 7f5deb954640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5de4146640 0x7f5de41e0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.851+0000 7f5deb954640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5de4151cd0 con 0x7f5de40acf70 2024-04-25T19:44:17.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.851+0000 7f5deb954640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5de4151b50 con 0x7f5de4146640 2024-04-25T19:44:17.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.851+0000 7f5deb954640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f5de4151e50 con 0x7f5de406df40 2024-04-25T19:44:17.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.851+0000 7f5de8eef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5de40acf70 0x7f5de413fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.851+0000 7f5de96f0640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5de406df40 0x7f5de41453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.852+0000 7f5de8eef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5de40acf70 0x7f5de413fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39720/0 (socket says 172.21.15.29:39720) 2024-04-25T19:44:17.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.852+0000 7f5de8eef640 1 -- 172.21.15.29:0/2791107948 learned_addr learned my addr 172.21.15.29:0/2791107948 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:17.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.852+0000 7f5de96f0640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5de406df40 0x7f5de41453c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:35088/0 (socket says 172.21.15.29:35088) 2024-04-25T19:44:17.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.852+0000 7f5de9ef1640 1 --2- 172.21.15.29:0/2791107948 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5de4146640 0x7f5de41e0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.852+0000 7f5de8eef640 1 -- 172.21.15.29:0/2791107948 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5de406df40 msgr2=0x7f5de41453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.852+0000 7f5de8eef640 1 --2- 172.21.15.29:0/2791107948 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5de406df40 0x7f5de41453c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.852+0000 7f5de8eef640 1 -- 172.21.15.29:0/2791107948 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5de4146640 msgr2=0x7f5de41e0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:17.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.852+0000 7f5de8eef640 1 --2- 172.21.15.29:0/2791107948 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5de4146640 0x7f5de41e0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:17.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.852+0000 7f5de8eef640 1 -- 172.21.15.29:0/2791107948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de4145c40 con 0x7f5de40acf70 2024-04-25T19:44:17.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.852+0000 7f5de9ef1640 1 --2- 172.21.15.29:0/2791107948 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5de4146640 0x7f5de41e0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:44:17.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.852+0000 7f5de8eef640 1 --2- 172.21.15.29:0/2791107948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5de40acf70 0x7f5de413fc30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800e9b0 tx=0x7f5dd800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:17.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.852+0000 7f5de1ffb640 1 -- 172.21.15.29:0/2791107948 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8017070 con 0x7f5de40acf70 2024-04-25T19:44:17.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.852+0000 7f5de1ffb640 1 -- 172.21.15.29:0/2791107948 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dd800c9a0 con 0x7f5de40acf70 2024-04-25T19:44:17.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.852+0000 7f5de1ffb640 1 -- 172.21.15.29:0/2791107948 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd800cd50 con 0x7f5de40acf70 2024-04-25T19:44:17.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.852+0000 7f5deb954640 1 -- 172.21.15.29:0/2791107948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5de413f220 con 0x7f5de40acf70 2024-04-25T19:44:17.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.852+0000 7f5deb954640 1 -- 172.21.15.29:0/2791107948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5de40b6470 con 0x7f5de40acf70 2024-04-25T19:44:17.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.854+0000 7f5deb954640 1 -- 172.21.15.29:0/2791107948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5de40507b0 con 0x7f5de40acf70 2024-04-25T19:44:17.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.855+0000 7f5de1ffb640 1 -- 172.21.15.29:0/2791107948 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f5dd801a070 con 0x7f5de40acf70 2024-04-25T19:44:17.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.855+0000 7f5de1ffb640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:17.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.856+0000 7f5de1ffb640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:17.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.856+0000 7f5de1ffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:17.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.856+0000 7f5de1ffb640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:17.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.856+0000 7f5de1ffb640 1 --2- 172.21.15.29:0/2791107948 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5dd00ac530 0x7f5dd00aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:17.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.856+0000 7f5de1ffb640 1 -- 172.21.15.29:0/2791107948 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f5dd80d9b30 con 0x7f5de40acf70 2024-04-25T19:44:17.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.856+0000 7f5de96f0640 1 --2- 172.21.15.29:0/2791107948 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5dd00ac530 0x7f5dd00aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:17.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.856+0000 7f5de96f0640 1 --2- 172.21.15.29:0/2791107948 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5dd00ac530 0x7f5dd00aea40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc000f60 tx=0x7f5ddc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:17.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:17.859+0000 7f5de1ffb640 1 -- 172.21.15.29:0/2791107948 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5dd80a4700 con 0x7f5de40acf70 2024-04-25T19:44:18.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.040+0000 7f5deb954640 20 mgrc start_command cmd: [{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}] 2024-04-25T19:44:18.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.040+0000 7f5deb954640 1 -- 172.21.15.29:0/2791107948 --> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5de40ba6f0 con 0x7f5dd00ac530 2024-04-25T19:44:18.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.048+0000 7f5de1ffb640 1 -- 172.21.15.29:0/2791107948 <== mgr.4108 v2:172.21.15.102:6824/1439704316 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+268533 (secure 0 0 0) 0x7f5de40ba6f0 con 0x7f5dd00ac530 2024-04-25T19:44:18.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.048+0000 7f5de1ffb640 20 mgrc handle_command_reply tid 0 r 0 2024-04-25T19:44:18.052 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:18.054 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-04-25T19:44:18.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.051+0000 7f5deb954640 10 mgrc shutdown 2024-04-25T19:44:18.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.051+0000 7f5deb954640 1 -- 172.21.15.29:0/2791107948 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5dd00ac530 msgr2=0x7f5dd00aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:18.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.051+0000 7f5deb954640 1 --2- 172.21.15.29:0/2791107948 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5dd00ac530 0x7f5dd00aea40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc000f60 tx=0x7f5ddc009040 comp rx=0 tx=0).stop 2024-04-25T19:44:18.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.051+0000 7f5deb954640 1 -- 172.21.15.29:0/2791107948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5de40acf70 msgr2=0x7f5de413fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:18.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.051+0000 7f5deb954640 1 --2- 172.21.15.29:0/2791107948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5de40acf70 0x7f5de413fc30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800e9b0 tx=0x7f5dd800ee70 comp rx=0 tx=0).stop 2024-04-25T19:44:18.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.051+0000 7f5deb954640 1 -- 172.21.15.29:0/2791107948 shutdown_connections 2024-04-25T19:44:18.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.051+0000 7f5deb954640 1 --2- 172.21.15.29:0/2791107948 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5dd00ac530 0x7f5dd00aea40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.051+0000 7f5deb954640 1 --2- 172.21.15.29:0/2791107948 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5de4146640 0x7f5de41e0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.051+0000 7f5deb954640 1 --2- 172.21.15.29:0/2791107948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5de40acf70 0x7f5de413fc30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.051+0000 7f5deb954640 1 --2- 172.21.15.29:0/2791107948 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5de406df40 0x7f5de41453c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.051+0000 7f5deb954640 1 -- 172.21.15.29:0/2791107948 >> 172.21.15.29:0/2791107948 conn(0x7f5de40ab710 msgr2=0x7f5de40a5540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:18.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.051+0000 7f5deb954640 1 -- 172.21.15.29:0/2791107948 shutdown_connections 2024-04-25T19:44:18.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.052+0000 7f5deb954640 1 -- 172.21.15.29:0/2791107948 wait complete. 2024-04-25T19:44:18.064 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-04-25T19:44:16.083590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":114192,"kb_used_data":1944,"kb_used_omap":6,"kb_used_meta":112185,"kb_avail":377373168,"statfs":{"total":386547056640,"available":386430124032,"internally_reserved":0,"allocated":1990656,"data_stored":1436532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":114878230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.055105"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043136+0000","last_change":"2024-04-25T19:44:10.065844+0000","last_active":"2024-04-25T19:44:11.043136+0000","last_peered":"2024-04-25T19:44:11.043136+0000","last_clean":"2024-04-25T19:44:11.043136+0000","last_became_active":"2024-04-25T19:44:10.065543+0000","last_became_peered":"2024-04-25T19:44:10.065543+0000","last_unstale":"2024-04-25T19:44:11.043136+0000","last_undegraded":"2024-04-25T19:44:11.043136+0000","last_fullsized":"2024-04-25T19:44:11.043136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:28.992782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042952+0000","last_change":"2024-04-25T19:44:10.054974+0000","last_active":"2024-04-25T19:44:11.042952+0000","last_peered":"2024-04-25T19:44:11.042952+0000","last_clean":"2024-04-25T19:44:11.042952+0000","last_became_active":"2024-04-25T19:44:10.054042+0000","last_became_peered":"2024-04-25T19:44:10.054042+0000","last_unstale":"2024-04-25T19:44:11.042952+0000","last_undegraded":"2024-04-25T19:44:11.042952+0000","last_fullsized":"2024-04-25T19:44:11.042952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:56.154645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436639+0000","last_change":"2024-04-25T19:44:10.060234+0000","last_active":"2024-04-25T19:44:11.436639+0000","last_peered":"2024-04-25T19:44:11.436639+0000","last_clean":"2024-04-25T19:44:11.436639+0000","last_became_active":"2024-04-25T19:44:10.059094+0000","last_became_peered":"2024-04-25T19:44:10.059094+0000","last_unstale":"2024-04-25T19:44:11.436639+0000","last_undegraded":"2024-04-25T19:44:11.436639+0000","last_fullsized":"2024-04-25T19:44:11.436639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:18.383491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745013+0000","last_change":"2024-04-25T19:44:10.056978+0000","last_active":"2024-04-25T19:44:11.745013+0000","last_peered":"2024-04-25T19:44:11.745013+0000","last_clean":"2024-04-25T19:44:11.745013+0000","last_became_active":"2024-04-25T19:44:10.051835+0000","last_became_peered":"2024-04-25T19:44:10.051835+0000","last_unstale":"2024-04-25T19:44:11.745013+0000","last_undegraded":"2024-04-25T19:44:11.745013+0000","last_fullsized":"2024-04-25T19:44:11.745013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:05:36.546123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436228+0000","last_change":"2024-04-25T19:44:10.058920+0000","last_active":"2024-04-25T19:44:11.436228+0000","last_peered":"2024-04-25T19:44:11.436228+0000","last_clean":"2024-04-25T19:44:11.436228+0000","last_became_active":"2024-04-25T19:44:10.058585+0000","last_became_peered":"2024-04-25T19:44:10.058585+0000","last_unstale":"2024-04-25T19:44:11.436228+0000","last_undegraded":"2024-04-25T19:44:11.436228+0000","last_fullsized":"2024-04-25T19:44:11.436228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:48:06.055133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435841+0000","last_change":"2024-04-25T19:44:10.055592+0000","last_active":"2024-04-25T19:44:11.435841+0000","last_peered":"2024-04-25T19:44:11.435841+0000","last_clean":"2024-04-25T19:44:11.435841+0000","last_became_active":"2024-04-25T19:44:10.053926+0000","last_became_peered":"2024-04-25T19:44:10.053926+0000","last_unstale":"2024-04-25T19:44:11.435841+0000","last_undegraded":"2024-04-25T19:44:11.435841+0000","last_fullsized":"2024-04-25T19:44:11.435841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:01.889093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435699+0000","last_change":"2024-04-25T19:44:10.058739+0000","last_active":"2024-04-25T19:44:11.435699+0000","last_peered":"2024-04-25T19:44:11.435699+0000","last_clean":"2024-04-25T19:44:11.435699+0000","last_became_active":"2024-04-25T19:44:10.053725+0000","last_became_peered":"2024-04-25T19:44:10.053725+0000","last_unstale":"2024-04-25T19:44:11.435699+0000","last_undegraded":"2024-04-25T19:44:11.435699+0000","last_fullsized":"2024-04-25T19:44:11.435699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:49.082468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745508+0000","last_change":"2024-04-25T19:44:10.069350+0000","last_active":"2024-04-25T19:44:11.745508+0000","last_peered":"2024-04-25T19:44:11.745508+0000","last_clean":"2024-04-25T19:44:11.745508+0000","last_became_active":"2024-04-25T19:44:10.069065+0000","last_became_peered":"2024-04-25T19:44:10.069065+0000","last_unstale":"2024-04-25T19:44:11.745508+0000","last_undegraded":"2024-04-25T19:44:11.745508+0000","last_fullsized":"2024-04-25T19:44:11.745508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:43:06.731936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042767+0000","last_change":"2024-04-25T19:44:10.052909+0000","last_active":"2024-04-25T19:44:11.042767+0000","last_peered":"2024-04-25T19:44:11.042767+0000","last_clean":"2024-04-25T19:44:11.042767+0000","last_became_active":"2024-04-25T19:44:10.052592+0000","last_became_peered":"2024-04-25T19:44:10.052592+0000","last_unstale":"2024-04-25T19:44:11.042767+0000","last_undegraded":"2024-04-25T19:44:11.042767+0000","last_fullsized":"2024-04-25T19:44:11.042767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:50.564937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745396+0000","last_change":"2024-04-25T19:44:10.060703+0000","last_active":"2024-04-25T19:44:11.745396+0000","last_peered":"2024-04-25T19:44:11.745396+0000","last_clean":"2024-04-25T19:44:11.745396+0000","last_became_active":"2024-04-25T19:44:10.058013+0000","last_became_peered":"2024-04-25T19:44:10.058013+0000","last_unstale":"2024-04-25T19:44:11.745396+0000","last_undegraded":"2024-04-25T19:44:11.745396+0000","last_fullsized":"2024-04-25T19:44:11.745396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:15:17.871931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745517+0000","last_change":"2024-04-25T19:44:10.049572+0000","last_active":"2024-04-25T19:44:11.745517+0000","last_peered":"2024-04-25T19:44:11.745517+0000","last_clean":"2024-04-25T19:44:11.745517+0000","last_became_active":"2024-04-25T19:44:10.049249+0000","last_became_peered":"2024-04-25T19:44:10.049249+0000","last_unstale":"2024-04-25T19:44:11.745517+0000","last_undegraded":"2024-04-25T19:44:11.745517+0000","last_fullsized":"2024-04-25T19:44:11.745517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:52:09.103201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043197+0000","last_change":"2024-04-25T19:44:10.056274+0000","last_active":"2024-04-25T19:44:11.043197+0000","last_peered":"2024-04-25T19:44:11.043197+0000","last_clean":"2024-04-25T19:44:11.043197+0000","last_became_active":"2024-04-25T19:44:10.055981+0000","last_became_peered":"2024-04-25T19:44:10.055981+0000","last_unstale":"2024-04-25T19:44:11.043197+0000","last_undegraded":"2024-04-25T19:44:11.043197+0000","last_fullsized":"2024-04-25T19:44:11.043197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:52:52.432869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042230+0000","last_change":"2024-04-25T19:44:10.044363+0000","last_active":"2024-04-25T19:44:11.042230+0000","last_peered":"2024-04-25T19:44:11.042230+0000","last_clean":"2024-04-25T19:44:11.042230+0000","last_became_active":"2024-04-25T19:44:10.044033+0000","last_became_peered":"2024-04-25T19:44:10.044033+0000","last_unstale":"2024-04-25T19:44:11.042230+0000","last_undegraded":"2024-04-25T19:44:11.042230+0000","last_fullsized":"2024-04-25T19:44:11.042230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:19.654106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.744782+0000","last_change":"2024-04-25T19:44:10.068850+0000","last_active":"2024-04-25T19:44:11.744782+0000","last_peered":"2024-04-25T19:44:11.744782+0000","last_clean":"2024-04-25T19:44:11.744782+0000","last_became_active":"2024-04-25T19:44:10.068565+0000","last_became_peered":"2024-04-25T19:44:10.068565+0000","last_unstale":"2024-04-25T19:44:11.744782+0000","last_undegraded":"2024-04-25T19:44:11.744782+0000","last_fullsized":"2024-04-25T19:44:11.744782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:40.201502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.744781+0000","last_change":"2024-04-25T19:44:10.056683+0000","last_active":"2024-04-25T19:44:11.744781+0000","last_peered":"2024-04-25T19:44:11.744781+0000","last_clean":"2024-04-25T19:44:11.744781+0000","last_became_active":"2024-04-25T19:44:10.056375+0000","last_became_peered":"2024-04-25T19:44:10.056375+0000","last_unstale":"2024-04-25T19:44:11.744781+0000","last_undegraded":"2024-04-25T19:44:11.744781+0000","last_fullsized":"2024-04-25T19:44:11.744781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:50:21.528308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435257+0000","last_change":"2024-04-25T19:44:10.056384+0000","last_active":"2024-04-25T19:44:11.435257+0000","last_peered":"2024-04-25T19:44:11.435257+0000","last_clean":"2024-04-25T19:44:11.435257+0000","last_became_active":"2024-04-25T19:44:10.054724+0000","last_became_peered":"2024-04-25T19:44:10.054724+0000","last_unstale":"2024-04-25T19:44:11.435257+0000","last_undegraded":"2024-04-25T19:44:11.435257+0000","last_fullsized":"2024-04-25T19:44:11.435257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:46:59.212591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745764+0000","last_change":"2024-04-25T19:44:10.069348+0000","last_active":"2024-04-25T19:44:11.745764+0000","last_peered":"2024-04-25T19:44:11.745764+0000","last_clean":"2024-04-25T19:44:11.745764+0000","last_became_active":"2024-04-25T19:44:10.068699+0000","last_became_peered":"2024-04-25T19:44:10.068699+0000","last_unstale":"2024-04-25T19:44:11.745764+0000","last_undegraded":"2024-04-25T19:44:11.745764+0000","last_fullsized":"2024-04-25T19:44:11.745764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:33:58.495931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436501+0000","last_change":"2024-04-25T19:44:10.061949+0000","last_active":"2024-04-25T19:44:11.436501+0000","last_peered":"2024-04-25T19:44:11.436501+0000","last_clean":"2024-04-25T19:44:11.436501+0000","last_became_active":"2024-04-25T19:44:10.061187+0000","last_became_peered":"2024-04-25T19:44:10.061187+0000","last_unstale":"2024-04-25T19:44:11.436501+0000","last_undegraded":"2024-04-25T19:44:11.436501+0000","last_fullsized":"2024-04-25T19:44:11.436501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:41:07.670796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042266+0000","last_change":"2024-04-25T19:44:10.045506+0000","last_active":"2024-04-25T19:44:11.042266+0000","last_peered":"2024-04-25T19:44:11.042266+0000","last_clean":"2024-04-25T19:44:11.042266+0000","last_became_active":"2024-04-25T19:44:10.044359+0000","last_became_peered":"2024-04-25T19:44:10.044359+0000","last_unstale":"2024-04-25T19:44:11.042266+0000","last_undegraded":"2024-04-25T19:44:11.042266+0000","last_fullsized":"2024-04-25T19:44:11.042266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:43:23.290058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745208+0000","last_change":"2024-04-25T19:44:10.051944+0000","last_active":"2024-04-25T19:44:11.745208+0000","last_peered":"2024-04-25T19:44:11.745208+0000","last_clean":"2024-04-25T19:44:11.745208+0000","last_became_active":"2024-04-25T19:44:10.051469+0000","last_became_peered":"2024-04-25T19:44:10.051469+0000","last_unstale":"2024-04-25T19:44:11.745208+0000","last_undegraded":"2024-04-25T19:44:11.745208+0000","last_fullsized":"2024-04-25T19:44:11.745208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:46:54.526883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745057+0000","last_change":"2024-04-25T19:44:10.060425+0000","last_active":"2024-04-25T19:44:11.745057+0000","last_peered":"2024-04-25T19:44:11.745057+0000","last_clean":"2024-04-25T19:44:11.745057+0000","last_became_active":"2024-04-25T19:44:10.059664+0000","last_became_peered":"2024-04-25T19:44:10.059664+0000","last_unstale":"2024-04-25T19:44:11.745057+0000","last_undegraded":"2024-04-25T19:44:11.745057+0000","last_fullsized":"2024-04-25T19:44:11.745057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:10:11.581750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.744779+0000","last_change":"2024-04-25T19:44:10.068792+0000","last_active":"2024-04-25T19:44:11.744779+0000","last_peered":"2024-04-25T19:44:11.744779+0000","last_clean":"2024-04-25T19:44:11.744779+0000","last_became_active":"2024-04-25T19:44:10.068477+0000","last_became_peered":"2024-04-25T19:44:10.068477+0000","last_unstale":"2024-04-25T19:44:11.744779+0000","last_undegraded":"2024-04-25T19:44:11.744779+0000","last_fullsized":"2024-04-25T19:44:11.744779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:18.399916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435517+0000","last_change":"2024-04-25T19:44:10.058966+0000","last_active":"2024-04-25T19:44:11.435517+0000","last_peered":"2024-04-25T19:44:11.435517+0000","last_clean":"2024-04-25T19:44:11.435517+0000","last_became_active":"2024-04-25T19:44:10.058083+0000","last_became_peered":"2024-04-25T19:44:10.058083+0000","last_unstale":"2024-04-25T19:44:11.435517+0000","last_undegraded":"2024-04-25T19:44:11.435517+0000","last_fullsized":"2024-04-25T19:44:11.435517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:34.810052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745205+0000","last_change":"2024-04-25T19:44:10.057670+0000","last_active":"2024-04-25T19:44:11.745205+0000","last_peered":"2024-04-25T19:44:11.745205+0000","last_clean":"2024-04-25T19:44:11.745205+0000","last_became_active":"2024-04-25T19:44:10.057360+0000","last_became_peered":"2024-04-25T19:44:10.057360+0000","last_unstale":"2024-04-25T19:44:11.745205+0000","last_undegraded":"2024-04-25T19:44:11.745205+0000","last_fullsized":"2024-04-25T19:44:11.745205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:18:02.918434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745808+0000","last_change":"2024-04-25T19:44:10.069281+0000","last_active":"2024-04-25T19:44:11.745808+0000","last_peered":"2024-04-25T19:44:11.745808+0000","last_clean":"2024-04-25T19:44:11.745808+0000","last_became_active":"2024-04-25T19:44:10.068327+0000","last_became_peered":"2024-04-25T19:44:10.068327+0000","last_unstale":"2024-04-25T19:44:11.745808+0000","last_undegraded":"2024-04-25T19:44:11.745808+0000","last_fullsized":"2024-04-25T19:44:11.745808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:46:59.212591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042716+0000","last_change":"2024-04-25T19:44:10.055433+0000","last_active":"2024-04-25T19:44:11.042716+0000","last_peered":"2024-04-25T19:44:11.042716+0000","last_clean":"2024-04-25T19:44:11.042716+0000","last_became_active":"2024-04-25T19:44:10.055058+0000","last_became_peered":"2024-04-25T19:44:10.055058+0000","last_unstale":"2024-04-25T19:44:11.042716+0000","last_undegraded":"2024-04-25T19:44:11.042716+0000","last_fullsized":"2024-04-25T19:44:11.042716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:56.154645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.437038+0000","last_change":"2024-04-25T19:44:10.064610+0000","last_active":"2024-04-25T19:44:11.437038+0000","last_peered":"2024-04-25T19:44:11.437038+0000","last_clean":"2024-04-25T19:44:11.437038+0000","last_became_active":"2024-04-25T19:44:10.064295+0000","last_became_peered":"2024-04-25T19:44:10.064295+0000","last_unstale":"2024-04-25T19:44:11.437038+0000","last_undegraded":"2024-04-25T19:44:11.437038+0000","last_fullsized":"2024-04-25T19:44:11.437038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:09.492760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043265+0000","last_change":"2024-04-25T19:44:10.050485+0000","last_active":"2024-04-25T19:44:11.043265+0000","last_peered":"2024-04-25T19:44:11.043265+0000","last_clean":"2024-04-25T19:44:11.043265+0000","last_became_active":"2024-04-25T19:44:10.050141+0000","last_became_peered":"2024-04-25T19:44:10.050141+0000","last_unstale":"2024-04-25T19:44:11.043265+0000","last_undegraded":"2024-04-25T19:44:11.043265+0000","last_fullsized":"2024-04-25T19:44:11.043265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:40.201502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042339+0000","last_change":"2024-04-25T19:44:10.044738+0000","last_active":"2024-04-25T19:44:11.042339+0000","last_peered":"2024-04-25T19:44:11.042339+0000","last_clean":"2024-04-25T19:44:11.042339+0000","last_became_active":"2024-04-25T19:44:10.044326+0000","last_became_peered":"2024-04-25T19:44:10.044326+0000","last_unstale":"2024-04-25T19:44:11.042339+0000","last_undegraded":"2024-04-25T19:44:11.042339+0000","last_fullsized":"2024-04-25T19:44:11.042339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:21.909648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745442+0000","last_change":"2024-04-25T19:44:10.062939+0000","last_active":"2024-04-25T19:44:11.745442+0000","last_peered":"2024-04-25T19:44:11.745442+0000","last_clean":"2024-04-25T19:44:11.745442+0000","last_became_active":"2024-04-25T19:44:10.057793+0000","last_became_peered":"2024-04-25T19:44:10.057793+0000","last_unstale":"2024-04-25T19:44:11.745442+0000","last_undegraded":"2024-04-25T19:44:11.745442+0000","last_fullsized":"2024-04-25T19:44:11.745442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:30.311117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.041761+0000","last_change":"2024-04-25T19:44:10.053059+0000","last_active":"2024-04-25T19:44:11.041761+0000","last_peered":"2024-04-25T19:44:11.041761+0000","last_clean":"2024-04-25T19:44:11.041761+0000","last_became_active":"2024-04-25T19:44:10.052494+0000","last_became_peered":"2024-04-25T19:44:10.052494+0000","last_unstale":"2024-04-25T19:44:11.041761+0000","last_undegraded":"2024-04-25T19:44:11.041761+0000","last_fullsized":"2024-04-25T19:44:11.041761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:38:53.994523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745521+0000","last_change":"2024-04-25T19:44:10.060010+0000","last_active":"2024-04-25T19:44:11.745521+0000","last_peered":"2024-04-25T19:44:11.745521+0000","last_clean":"2024-04-25T19:44:11.745521+0000","last_became_active":"2024-04-25T19:44:10.059598+0000","last_became_peered":"2024-04-25T19:44:10.059598+0000","last_unstale":"2024-04-25T19:44:11.745521+0000","last_undegraded":"2024-04-25T19:44:11.745521+0000","last_fullsized":"2024-04-25T19:44:11.745521+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:40.247386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435938+0000","last_change":"2024-04-25T19:44:10.059147+0000","last_active":"2024-04-25T19:44:11.435938+0000","last_peered":"2024-04-25T19:44:11.435938+0000","last_clean":"2024-04-25T19:44:11.435938+0000","last_became_active":"2024-04-25T19:44:10.058363+0000","last_became_peered":"2024-04-25T19:44:10.058363+0000","last_unstale":"2024-04-25T19:44:11.435938+0000","last_undegraded":"2024-04-25T19:44:11.435938+0000","last_fullsized":"2024-04-25T19:44:11.435938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:05:57.081479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436415+0000","last_change":"2024-04-25T19:44:09.056791+0000","last_active":"2024-04-25T19:44:11.436415+0000","last_peered":"2024-04-25T19:44:11.436415+0000","last_clean":"2024-04-25T19:44:11.436415+0000","last_became_active":"2024-04-25T19:44:09.056430+0000","last_became_peered":"2024-04-25T19:44:09.056430+0000","last_unstale":"2024-04-25T19:44:11.436415+0000","last_undegraded":"2024-04-25T19:44:11.436415+0000","last_fullsized":"2024-04-25T19:44:11.436415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:18.652125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436777+0000","last_change":"2024-04-25T19:44:10.062414+0000","last_active":"2024-04-25T19:44:11.436777+0000","last_peered":"2024-04-25T19:44:11.436777+0000","last_clean":"2024-04-25T19:44:11.436777+0000","last_became_active":"2024-04-25T19:44:10.061691+0000","last_became_peered":"2024-04-25T19:44:10.061691+0000","last_unstale":"2024-04-25T19:44:11.436777+0000","last_undegraded":"2024-04-25T19:44:11.436777+0000","last_fullsized":"2024-04-25T19:44:11.436777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:29:27.722435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043208+0000","last_change":"2024-04-25T19:44:09.052305+0000","last_active":"2024-04-25T19:44:11.043208+0000","last_peered":"2024-04-25T19:44:11.043208+0000","last_clean":"2024-04-25T19:44:11.043208+0000","last_became_active":"2024-04-25T19:44:09.051968+0000","last_became_peered":"2024-04-25T19:44:09.051968+0000","last_unstale":"2024-04-25T19:44:11.043208+0000","last_undegraded":"2024-04-25T19:44:11.043208+0000","last_fullsized":"2024-04-25T19:44:11.043208+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:11:22.034588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436495+0000","last_change":"2024-04-25T19:44:10.062199+0000","last_active":"2024-04-25T19:44:11.436495+0000","last_peered":"2024-04-25T19:44:11.436495+0000","last_clean":"2024-04-25T19:44:11.436495+0000","last_became_active":"2024-04-25T19:44:10.061613+0000","last_became_peered":"2024-04-25T19:44:10.061613+0000","last_unstale":"2024-04-25T19:44:11.436495+0000","last_undegraded":"2024-04-25T19:44:11.436495+0000","last_fullsized":"2024-04-25T19:44:11.436495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:56:49.470340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436342+0000","last_change":"2024-04-25T19:44:09.058706+0000","last_active":"2024-04-25T19:44:11.436342+0000","last_peered":"2024-04-25T19:44:11.436342+0000","last_clean":"2024-04-25T19:44:11.436342+0000","last_became_active":"2024-04-25T19:44:09.057009+0000","last_became_peered":"2024-04-25T19:44:09.057009+0000","last_unstale":"2024-04-25T19:44:11.436342+0000","last_undegraded":"2024-04-25T19:44:11.436342+0000","last_fullsized":"2024-04-25T19:44:11.436342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:17:26.264206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043125+0000","last_change":"2024-04-25T19:44:10.058621+0000","last_active":"2024-04-25T19:44:11.043125+0000","last_peered":"2024-04-25T19:44:11.043125+0000","last_clean":"2024-04-25T19:44:11.043125+0000","last_became_active":"2024-04-25T19:44:10.058237+0000","last_became_peered":"2024-04-25T19:44:10.058237+0000","last_unstale":"2024-04-25T19:44:11.043125+0000","last_undegraded":"2024-04-25T19:44:11.043125+0000","last_fullsized":"2024-04-25T19:44:11.043125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:18.399916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745612+0000","last_change":"2024-04-25T19:44:09.054405+0000","last_active":"2024-04-25T19:44:11.745612+0000","last_peered":"2024-04-25T19:44:11.745612+0000","last_clean":"2024-04-25T19:44:11.745612+0000","last_became_active":"2024-04-25T19:44:09.054065+0000","last_became_peered":"2024-04-25T19:44:09.054065+0000","last_unstale":"2024-04-25T19:44:11.745612+0000","last_undegraded":"2024-04-25T19:44:11.745612+0000","last_fullsized":"2024-04-25T19:44:11.745612+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:59.352345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043210+0000","last_change":"2024-04-25T19:44:09.047923+0000","last_active":"2024-04-25T19:44:11.043210+0000","last_peered":"2024-04-25T19:44:11.043210+0000","last_clean":"2024-04-25T19:44:11.043210+0000","last_became_active":"2024-04-25T19:44:09.047605+0000","last_became_peered":"2024-04-25T19:44:09.047605+0000","last_unstale":"2024-04-25T19:44:11.043210+0000","last_undegraded":"2024-04-25T19:44:11.043210+0000","last_fullsized":"2024-04-25T19:44:11.043210+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:30:43.807694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436073+0000","last_change":"2024-04-25T19:44:10.067620+0000","last_active":"2024-04-25T19:44:11.436073+0000","last_peered":"2024-04-25T19:44:11.436073+0000","last_clean":"2024-04-25T19:44:11.436073+0000","last_became_active":"2024-04-25T19:44:10.066364+0000","last_became_peered":"2024-04-25T19:44:10.066364+0000","last_unstale":"2024-04-25T19:44:11.436073+0000","last_undegraded":"2024-04-25T19:44:11.436073+0000","last_fullsized":"2024-04-25T19:44:11.436073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:20:30.629784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043379+0000","last_change":"2024-04-25T19:44:09.045871+0000","last_active":"2024-04-25T19:44:11.043379+0000","last_peered":"2024-04-25T19:44:11.043379+0000","last_clean":"2024-04-25T19:44:11.043379+0000","last_became_active":"2024-04-25T19:44:09.045017+0000","last_became_peered":"2024-04-25T19:44:09.045017+0000","last_unstale":"2024-04-25T19:44:11.043379+0000","last_undegraded":"2024-04-25T19:44:11.043379+0000","last_fullsized":"2024-04-25T19:44:11.043379+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:51:27.578108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.744790+0000","last_change":"2024-04-25T19:44:10.049695+0000","last_active":"2024-04-25T19:44:11.744790+0000","last_peered":"2024-04-25T19:44:11.744790+0000","last_clean":"2024-04-25T19:44:11.744790+0000","last_became_active":"2024-04-25T19:44:10.049361+0000","last_became_peered":"2024-04-25T19:44:10.049361+0000","last_unstale":"2024-04-25T19:44:11.744790+0000","last_undegraded":"2024-04-25T19:44:11.744790+0000","last_fullsized":"2024-04-25T19:44:11.744790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:57.562802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436867+0000","last_change":"2024-04-25T19:44:09.059742+0000","last_active":"2024-04-25T19:44:11.436867+0000","last_peered":"2024-04-25T19:44:11.436867+0000","last_clean":"2024-04-25T19:44:11.436867+0000","last_became_active":"2024-04-25T19:44:09.059318+0000","last_became_peered":"2024-04-25T19:44:09.059318+0000","last_unstale":"2024-04-25T19:44:11.436867+0000","last_undegraded":"2024-04-25T19:44:11.436867+0000","last_fullsized":"2024-04-25T19:44:11.436867+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:55.152664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043138+0000","last_change":"2024-04-25T19:44:10.056446+0000","last_active":"2024-04-25T19:44:11.043138+0000","last_peered":"2024-04-25T19:44:11.043138+0000","last_clean":"2024-04-25T19:44:11.043138+0000","last_became_active":"2024-04-25T19:44:10.056096+0000","last_became_peered":"2024-04-25T19:44:10.056096+0000","last_unstale":"2024-04-25T19:44:11.043138+0000","last_undegraded":"2024-04-25T19:44:11.043138+0000","last_fullsized":"2024-04-25T19:44:11.043138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:26:17.991276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436305+0000","last_change":"2024-04-25T19:44:09.054439+0000","last_active":"2024-04-25T19:44:11.436305+0000","last_peered":"2024-04-25T19:44:11.436305+0000","last_clean":"2024-04-25T19:44:11.436305+0000","last_became_active":"2024-04-25T19:44:09.053941+0000","last_became_peered":"2024-04-25T19:44:09.053941+0000","last_unstale":"2024-04-25T19:44:11.436305+0000","last_undegraded":"2024-04-25T19:44:11.436305+0000","last_fullsized":"2024-04-25T19:44:11.436305+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:55:44.329801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043164+0000","last_change":"2024-04-25T19:44:10.047455+0000","last_active":"2024-04-25T19:44:11.043164+0000","last_peered":"2024-04-25T19:44:11.043164+0000","last_clean":"2024-04-25T19:44:11.043164+0000","last_became_active":"2024-04-25T19:44:10.046766+0000","last_became_peered":"2024-04-25T19:44:10.046766+0000","last_unstale":"2024-04-25T19:44:11.043164+0000","last_undegraded":"2024-04-25T19:44:11.043164+0000","last_fullsized":"2024-04-25T19:44:11.043164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:46.058675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042351+0000","last_change":"2024-04-25T19:44:09.053423+0000","last_active":"2024-04-25T19:44:11.042351+0000","last_peered":"2024-04-25T19:44:11.042351+0000","last_clean":"2024-04-25T19:44:11.042351+0000","last_became_active":"2024-04-25T19:44:09.053138+0000","last_became_peered":"2024-04-25T19:44:09.053138+0000","last_unstale":"2024-04-25T19:44:11.042351+0000","last_undegraded":"2024-04-25T19:44:11.042351+0000","last_fullsized":"2024-04-25T19:44:11.042351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:11:22.034588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745419+0000","last_change":"2024-04-25T19:44:10.061890+0000","last_active":"2024-04-25T19:44:11.745419+0000","last_peered":"2024-04-25T19:44:11.745419+0000","last_clean":"2024-04-25T19:44:11.745419+0000","last_became_active":"2024-04-25T19:44:10.061525+0000","last_became_peered":"2024-04-25T19:44:10.061525+0000","last_unstale":"2024-04-25T19:44:11.745419+0000","last_undegraded":"2024-04-25T19:44:11.745419+0000","last_fullsized":"2024-04-25T19:44:11.745419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:33:29.948604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435794+0000","last_change":"2024-04-25T19:44:09.058395+0000","last_active":"2024-04-25T19:44:11.435794+0000","last_peered":"2024-04-25T19:44:11.435794+0000","last_clean":"2024-04-25T19:44:11.435794+0000","last_became_active":"2024-04-25T19:44:09.057411+0000","last_became_peered":"2024-04-25T19:44:09.057411+0000","last_unstale":"2024-04-25T19:44:11.435794+0000","last_undegraded":"2024-04-25T19:44:11.435794+0000","last_fullsized":"2024-04-25T19:44:11.435794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:53:44.798992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436721+0000","last_change":"2024-04-25T19:44:10.062688+0000","last_active":"2024-04-25T19:44:11.436721+0000","last_peered":"2024-04-25T19:44:11.436721+0000","last_clean":"2024-04-25T19:44:11.436721+0000","last_became_active":"2024-04-25T19:44:10.059124+0000","last_became_peered":"2024-04-25T19:44:10.059124+0000","last_unstale":"2024-04-25T19:44:11.436721+0000","last_undegraded":"2024-04-25T19:44:11.436721+0000","last_fullsized":"2024-04-25T19:44:11.436721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:18:06.928672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436491+0000","last_change":"2024-04-25T19:44:09.056965+0000","last_active":"2024-04-25T19:44:11.436491+0000","last_peered":"2024-04-25T19:44:11.436491+0000","last_clean":"2024-04-25T19:44:11.436491+0000","last_became_active":"2024-04-25T19:44:09.056557+0000","last_became_peered":"2024-04-25T19:44:09.056557+0000","last_unstale":"2024-04-25T19:44:11.436491+0000","last_undegraded":"2024-04-25T19:44:11.436491+0000","last_fullsized":"2024-04-25T19:44:11.436491+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:08:50.165109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043306+0000","last_change":"2024-04-25T19:44:10.066632+0000","last_active":"2024-04-25T19:44:11.043306+0000","last_peered":"2024-04-25T19:44:11.043306+0000","last_clean":"2024-04-25T19:44:11.043306+0000","last_became_active":"2024-04-25T19:44:10.066329+0000","last_became_peered":"2024-04-25T19:44:10.066329+0000","last_unstale":"2024-04-25T19:44:11.043306+0000","last_undegraded":"2024-04-25T19:44:11.043306+0000","last_fullsized":"2024-04-25T19:44:11.043306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:18.399916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435984+0000","last_change":"2024-04-25T19:44:09.058583+0000","last_active":"2024-04-25T19:44:11.435984+0000","last_peered":"2024-04-25T19:44:11.435984+0000","last_clean":"2024-04-25T19:44:11.435984+0000","last_became_active":"2024-04-25T19:44:09.057289+0000","last_became_peered":"2024-04-25T19:44:09.057289+0000","last_unstale":"2024-04-25T19:44:11.435984+0000","last_undegraded":"2024-04-25T19:44:11.435984+0000","last_fullsized":"2024-04-25T19:44:11.435984+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:43:22.288077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745913+0000","last_change":"2024-04-25T19:44:10.069026+0000","last_active":"2024-04-25T19:44:11.745913+0000","last_peered":"2024-04-25T19:44:11.745913+0000","last_clean":"2024-04-25T19:44:11.745913+0000","last_became_active":"2024-04-25T19:44:10.067900+0000","last_became_peered":"2024-04-25T19:44:10.067900+0000","last_unstale":"2024-04-25T19:44:11.745913+0000","last_undegraded":"2024-04-25T19:44:11.745913+0000","last_fullsized":"2024-04-25T19:44:11.745913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:51.252629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'72","reported_seq":116,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745679+0000","last_change":"2024-04-25T19:44:00.045570+0000","last_active":"2024-04-25T19:44:11.745679+0000","last_peered":"2024-04-25T19:44:11.745679+0000","last_clean":"2024-04-25T19:44:11.745679+0000","last_became_active":"2024-04-25T19:44:00.044414+0000","last_became_peered":"2024-04-25T19:44:00.044414+0000","last_unstale":"2024-04-25T19:44:11.745679+0000","last_undegraded":"2024-04-25T19:44:11.745679+0000","last_fullsized":"2024-04-25T19:44:11.745679+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:43:58.854910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:43:58.854910+0000","last_clean_scrub_stamp":"2024-04-25T19:43:58.854910+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:36.813186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436422+0000","last_change":"2024-04-25T19:44:04.907674+0000","last_active":"2024-04-25T19:44:11.436422+0000","last_peered":"2024-04-25T19:44:11.436422+0000","last_clean":"2024-04-25T19:44:11.436422+0000","last_became_active":"2024-04-25T19:44:04.907259+0000","last_became_peered":"2024-04-25T19:44:04.907259+0000","last_unstale":"2024-04-25T19:44:11.436422+0000","last_undegraded":"2024-04-25T19:44:11.436422+0000","last_fullsized":"2024-04-25T19:44:11.436422+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:58.023878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436111+0000","last_change":"2024-04-25T19:44:09.056946+0000","last_active":"2024-04-25T19:44:11.436111+0000","last_peered":"2024-04-25T19:44:11.436111+0000","last_clean":"2024-04-25T19:44:11.436111+0000","last_became_active":"2024-04-25T19:44:09.056544+0000","last_became_peered":"2024-04-25T19:44:09.056544+0000","last_unstale":"2024-04-25T19:44:11.436111+0000","last_undegraded":"2024-04-25T19:44:11.436111+0000","last_fullsized":"2024-04-25T19:44:11.436111+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:24:46.807744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436901+0000","last_change":"2024-04-25T19:44:10.060919+0000","last_active":"2024-04-25T19:44:11.436901+0000","last_peered":"2024-04-25T19:44:11.436901+0000","last_clean":"2024-04-25T19:44:11.436901+0000","last_became_active":"2024-04-25T19:44:10.059529+0000","last_became_peered":"2024-04-25T19:44:10.059529+0000","last_unstale":"2024-04-25T19:44:11.436901+0000","last_undegraded":"2024-04-25T19:44:11.436901+0000","last_fullsized":"2024-04-25T19:44:11.436901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:29.509488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436146+0000","last_change":"2024-04-25T19:44:04.904786+0000","last_active":"2024-04-25T19:44:11.436146+0000","last_peered":"2024-04-25T19:44:11.436146+0000","last_clean":"2024-04-25T19:44:11.436146+0000","last_became_active":"2024-04-25T19:44:04.904414+0000","last_became_peered":"2024-04-25T19:44:04.904414+0000","last_unstale":"2024-04-25T19:44:11.436146+0000","last_undegraded":"2024-04-25T19:44:11.436146+0000","last_fullsized":"2024-04-25T19:44:11.436146+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:48:36.719208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.041861+0000","last_change":"2024-04-25T19:44:09.040846+0000","last_active":"2024-04-25T19:44:11.041861+0000","last_peered":"2024-04-25T19:44:11.041861+0000","last_clean":"2024-04-25T19:44:11.041861+0000","last_became_active":"2024-04-25T19:44:09.040461+0000","last_became_peered":"2024-04-25T19:44:09.040461+0000","last_unstale":"2024-04-25T19:44:11.041861+0000","last_undegraded":"2024-04-25T19:44:11.041861+0000","last_fullsized":"2024-04-25T19:44:11.041861+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:48:47.178567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436718+0000","last_change":"2024-04-25T19:44:10.062165+0000","last_active":"2024-04-25T19:44:11.436718+0000","last_peered":"2024-04-25T19:44:11.436718+0000","last_clean":"2024-04-25T19:44:11.436718+0000","last_became_active":"2024-04-25T19:44:10.061626+0000","last_became_peered":"2024-04-25T19:44:10.061626+0000","last_unstale":"2024-04-25T19:44:11.436718+0000","last_undegraded":"2024-04-25T19:44:11.436718+0000","last_fullsized":"2024-04-25T19:44:11.436718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:03:01.981027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042646+0000","last_change":"2024-04-25T19:44:04.904405+0000","last_active":"2024-04-25T19:44:11.042646+0000","last_peered":"2024-04-25T19:44:11.042646+0000","last_clean":"2024-04-25T19:44:11.042646+0000","last_became_active":"2024-04-25T19:44:04.904045+0000","last_became_peered":"2024-04-25T19:44:04.904045+0000","last_unstale":"2024-04-25T19:44:11.042646+0000","last_undegraded":"2024-04-25T19:44:11.042646+0000","last_fullsized":"2024-04-25T19:44:11.042646+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:01:59.962699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.041684+0000","last_change":"2024-04-25T19:44:09.045036+0000","last_active":"2024-04-25T19:44:11.041684+0000","last_peered":"2024-04-25T19:44:11.041684+0000","last_clean":"2024-04-25T19:44:11.041684+0000","last_became_active":"2024-04-25T19:44:09.044686+0000","last_became_peered":"2024-04-25T19:44:09.044686+0000","last_unstale":"2024-04-25T19:44:11.041684+0000","last_undegraded":"2024-04-25T19:44:11.041684+0000","last_fullsized":"2024-04-25T19:44:11.041684+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:41:37.111120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042963+0000","last_change":"2024-04-25T19:44:10.053021+0000","last_active":"2024-04-25T19:44:11.042963+0000","last_peered":"2024-04-25T19:44:11.042963+0000","last_clean":"2024-04-25T19:44:11.042963+0000","last_became_active":"2024-04-25T19:44:10.052663+0000","last_became_peered":"2024-04-25T19:44:10.052663+0000","last_unstale":"2024-04-25T19:44:11.042963+0000","last_undegraded":"2024-04-25T19:44:11.042963+0000","last_fullsized":"2024-04-25T19:44:11.042963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:46.058675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436124+0000","last_change":"2024-04-25T19:44:04.907093+0000","last_active":"2024-04-25T19:44:11.436124+0000","last_peered":"2024-04-25T19:44:11.436124+0000","last_clean":"2024-04-25T19:44:11.436124+0000","last_became_active":"2024-04-25T19:44:04.906787+0000","last_became_peered":"2024-04-25T19:44:04.906787+0000","last_unstale":"2024-04-25T19:44:11.436124+0000","last_undegraded":"2024-04-25T19:44:11.436124+0000","last_fullsized":"2024-04-25T19:44:11.436124+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:28:59.374809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043241+0000","last_change":"2024-04-25T19:44:09.043484+0000","last_active":"2024-04-25T19:44:11.043241+0000","last_peered":"2024-04-25T19:44:11.043241+0000","last_clean":"2024-04-25T19:44:11.043241+0000","last_became_active":"2024-04-25T19:44:09.043155+0000","last_became_peered":"2024-04-25T19:44:09.043155+0000","last_unstale":"2024-04-25T19:44:11.043241+0000","last_undegraded":"2024-04-25T19:44:11.043241+0000","last_fullsized":"2024-04-25T19:44:11.043241+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:25.996441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043040+0000","last_change":"2024-04-25T19:44:10.050042+0000","last_active":"2024-04-25T19:44:11.043040+0000","last_peered":"2024-04-25T19:44:11.043040+0000","last_clean":"2024-04-25T19:44:11.043040+0000","last_became_active":"2024-04-25T19:44:10.049701+0000","last_became_peered":"2024-04-25T19:44:10.049701+0000","last_unstale":"2024-04-25T19:44:11.043040+0000","last_undegraded":"2024-04-25T19:44:11.043040+0000","last_fullsized":"2024-04-25T19:44:11.043040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:52:52.432869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043083+0000","last_change":"2024-04-25T19:44:09.040543+0000","last_active":"2024-04-25T19:44:11.043083+0000","last_peered":"2024-04-25T19:44:11.043083+0000","last_clean":"2024-04-25T19:44:11.043083+0000","last_became_active":"2024-04-25T19:44:09.040137+0000","last_became_peered":"2024-04-25T19:44:09.040137+0000","last_unstale":"2024-04-25T19:44:11.043083+0000","last_undegraded":"2024-04-25T19:44:11.043083+0000","last_fullsized":"2024-04-25T19:44:11.043083+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:59:23.033455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043278+0000","last_change":"2024-04-25T19:44:04.905598+0000","last_active":"2024-04-25T19:44:11.043278+0000","last_peered":"2024-04-25T19:44:11.043278+0000","last_clean":"2024-04-25T19:44:11.043278+0000","last_became_active":"2024-04-25T19:44:04.904778+0000","last_became_peered":"2024-04-25T19:44:04.904778+0000","last_unstale":"2024-04-25T19:44:11.043278+0000","last_undegraded":"2024-04-25T19:44:11.043278+0000","last_fullsized":"2024-04-25T19:44:11.043278+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:03:40.381610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745207+0000","last_change":"2024-04-25T19:44:10.063439+0000","last_active":"2024-04-25T19:44:11.745207+0000","last_peered":"2024-04-25T19:44:11.745207+0000","last_clean":"2024-04-25T19:44:11.745207+0000","last_became_active":"2024-04-25T19:44:10.062610+0000","last_became_peered":"2024-04-25T19:44:10.062610+0000","last_unstale":"2024-04-25T19:44:11.745207+0000","last_undegraded":"2024-04-25T19:44:11.745207+0000","last_fullsized":"2024-04-25T19:44:11.745207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:43:53.239881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043105+0000","last_change":"2024-04-25T19:44:09.044556+0000","last_active":"2024-04-25T19:44:11.043105+0000","last_peered":"2024-04-25T19:44:11.043105+0000","last_clean":"2024-04-25T19:44:11.043105+0000","last_became_active":"2024-04-25T19:44:09.044205+0000","last_became_peered":"2024-04-25T19:44:09.044205+0000","last_unstale":"2024-04-25T19:44:11.043105+0000","last_undegraded":"2024-04-25T19:44:11.043105+0000","last_fullsized":"2024-04-25T19:44:11.043105+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:59:23.033455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436963+0000","last_change":"2024-04-25T19:44:04.904842+0000","last_active":"2024-04-25T19:44:11.436963+0000","last_peered":"2024-04-25T19:44:11.436963+0000","last_clean":"2024-04-25T19:44:11.436963+0000","last_became_active":"2024-04-25T19:44:04.904449+0000","last_became_peered":"2024-04-25T19:44:04.904449+0000","last_unstale":"2024-04-25T19:44:11.436963+0000","last_undegraded":"2024-04-25T19:44:11.436963+0000","last_fullsized":"2024-04-25T19:44:11.436963+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:38.322685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.744965+0000","last_change":"2024-04-25T19:44:10.060164+0000","last_active":"2024-04-25T19:44:11.744965+0000","last_peered":"2024-04-25T19:44:11.744965+0000","last_clean":"2024-04-25T19:44:11.744965+0000","last_became_active":"2024-04-25T19:44:10.059272+0000","last_became_peered":"2024-04-25T19:44:10.059272+0000","last_unstale":"2024-04-25T19:44:11.744965+0000","last_undegraded":"2024-04-25T19:44:11.744965+0000","last_fullsized":"2024-04-25T19:44:11.744965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:49:57.885441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436380+0000","last_change":"2024-04-25T19:44:09.058552+0000","last_active":"2024-04-25T19:44:11.436380+0000","last_peered":"2024-04-25T19:44:11.436380+0000","last_clean":"2024-04-25T19:44:11.436380+0000","last_became_active":"2024-04-25T19:44:09.058107+0000","last_became_peered":"2024-04-25T19:44:09.058107+0000","last_unstale":"2024-04-25T19:44:11.436380+0000","last_undegraded":"2024-04-25T19:44:11.436380+0000","last_fullsized":"2024-04-25T19:44:11.436380+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:27.990801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042931+0000","last_change":"2024-04-25T19:44:04.904847+0000","last_active":"2024-04-25T19:44:11.042931+0000","last_peered":"2024-04-25T19:44:11.042931+0000","last_clean":"2024-04-25T19:44:11.042931+0000","last_became_active":"2024-04-25T19:44:04.904399+0000","last_became_peered":"2024-04-25T19:44:04.904399+0000","last_unstale":"2024-04-25T19:44:11.042931+0000","last_undegraded":"2024-04-25T19:44:11.042931+0000","last_fullsized":"2024-04-25T19:44:11.042931+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:31.155970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435636+0000","last_change":"2024-04-25T19:44:10.067351+0000","last_active":"2024-04-25T19:44:11.435636+0000","last_peered":"2024-04-25T19:44:11.435636+0000","last_clean":"2024-04-25T19:44:11.435636+0000","last_became_active":"2024-04-25T19:44:10.065833+0000","last_became_peered":"2024-04-25T19:44:10.065833+0000","last_unstale":"2024-04-25T19:44:11.435636+0000","last_undegraded":"2024-04-25T19:44:11.435636+0000","last_fullsized":"2024-04-25T19:44:11.435636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:41:44.189493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745703+0000","last_change":"2024-04-25T19:44:09.048708+0000","last_active":"2024-04-25T19:44:11.745703+0000","last_peered":"2024-04-25T19:44:11.745703+0000","last_clean":"2024-04-25T19:44:11.745703+0000","last_became_active":"2024-04-25T19:44:09.048320+0000","last_became_peered":"2024-04-25T19:44:09.048320+0000","last_unstale":"2024-04-25T19:44:11.745703+0000","last_undegraded":"2024-04-25T19:44:11.745703+0000","last_fullsized":"2024-04-25T19:44:11.745703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:48:47.178567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436571+0000","last_change":"2024-04-25T19:44:04.907462+0000","last_active":"2024-04-25T19:44:11.436571+0000","last_peered":"2024-04-25T19:44:11.436571+0000","last_clean":"2024-04-25T19:44:11.436571+0000","last_became_active":"2024-04-25T19:44:04.907134+0000","last_became_peered":"2024-04-25T19:44:04.907134+0000","last_unstale":"2024-04-25T19:44:11.436571+0000","last_undegraded":"2024-04-25T19:44:11.436571+0000","last_fullsized":"2024-04-25T19:44:11.436571+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:05:53.288205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042736+0000","last_change":"2024-04-25T19:44:10.062982+0000","last_active":"2024-04-25T19:44:11.042736+0000","last_peered":"2024-04-25T19:44:11.042736+0000","last_clean":"2024-04-25T19:44:11.042736+0000","last_became_active":"2024-04-25T19:44:10.062674+0000","last_became_peered":"2024-04-25T19:44:10.062674+0000","last_unstale":"2024-04-25T19:44:11.042736+0000","last_undegraded":"2024-04-25T19:44:11.042736+0000","last_fullsized":"2024-04-25T19:44:11.042736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:21.909648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043372+0000","last_change":"2024-04-25T19:44:10.056196+0000","last_active":"2024-04-25T19:44:11.043372+0000","last_peered":"2024-04-25T19:44:11.043372+0000","last_clean":"2024-04-25T19:44:11.043372+0000","last_became_active":"2024-04-25T19:44:10.055892+0000","last_became_peered":"2024-04-25T19:44:10.055892+0000","last_unstale":"2024-04-25T19:44:11.043372+0000","last_undegraded":"2024-04-25T19:44:11.043372+0000","last_fullsized":"2024-04-25T19:44:11.043372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:40.201502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436060+0000","last_change":"2024-04-25T19:44:09.056570+0000","last_active":"2024-04-25T19:44:11.436060+0000","last_peered":"2024-04-25T19:44:11.436060+0000","last_clean":"2024-04-25T19:44:11.436060+0000","last_became_active":"2024-04-25T19:44:09.055464+0000","last_became_peered":"2024-04-25T19:44:09.055464+0000","last_unstale":"2024-04-25T19:44:11.436060+0000","last_undegraded":"2024-04-25T19:44:11.436060+0000","last_fullsized":"2024-04-25T19:44:11.436060+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:48:00.496337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745172+0000","last_change":"2024-04-25T19:44:10.063199+0000","last_active":"2024-04-25T19:44:11.745172+0000","last_peered":"2024-04-25T19:44:11.745172+0000","last_clean":"2024-04-25T19:44:11.745172+0000","last_became_active":"2024-04-25T19:44:10.062127+0000","last_became_peered":"2024-04-25T19:44:10.062127+0000","last_unstale":"2024-04-25T19:44:11.745172+0000","last_undegraded":"2024-04-25T19:44:11.745172+0000","last_fullsized":"2024-04-25T19:44:11.745172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:36:33.677454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042922+0000","last_change":"2024-04-25T19:44:09.059522+0000","last_active":"2024-04-25T19:44:11.042922+0000","last_peered":"2024-04-25T19:44:11.042922+0000","last_clean":"2024-04-25T19:44:11.042922+0000","last_became_active":"2024-04-25T19:44:09.059182+0000","last_became_peered":"2024-04-25T19:44:09.059182+0000","last_unstale":"2024-04-25T19:44:11.042922+0000","last_undegraded":"2024-04-25T19:44:11.042922+0000","last_fullsized":"2024-04-25T19:44:11.042922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:25.699599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043346+0000","last_change":"2024-04-25T19:44:09.054779+0000","last_active":"2024-04-25T19:44:11.043346+0000","last_peered":"2024-04-25T19:44:11.043346+0000","last_clean":"2024-04-25T19:44:11.043346+0000","last_became_active":"2024-04-25T19:44:09.054451+0000","last_became_peered":"2024-04-25T19:44:09.054451+0000","last_unstale":"2024-04-25T19:44:11.043346+0000","last_undegraded":"2024-04-25T19:44:11.043346+0000","last_fullsized":"2024-04-25T19:44:11.043346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:59.352345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.041751+0000","last_change":"2024-04-25T19:44:10.043921+0000","last_active":"2024-04-25T19:44:11.041751+0000","last_peered":"2024-04-25T19:44:11.041751+0000","last_clean":"2024-04-25T19:44:11.041751+0000","last_became_active":"2024-04-25T19:44:10.043528+0000","last_became_peered":"2024-04-25T19:44:10.043528+0000","last_unstale":"2024-04-25T19:44:11.041751+0000","last_undegraded":"2024-04-25T19:44:11.041751+0000","last_fullsized":"2024-04-25T19:44:11.041751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:55:45.331783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745974+0000","last_change":"2024-04-25T19:44:09.055027+0000","last_active":"2024-04-25T19:44:11.745974+0000","last_peered":"2024-04-25T19:44:11.745974+0000","last_clean":"2024-04-25T19:44:11.745974+0000","last_became_active":"2024-04-25T19:44:09.054691+0000","last_became_peered":"2024-04-25T19:44:09.054691+0000","last_unstale":"2024-04-25T19:44:11.745974+0000","last_undegraded":"2024-04-25T19:44:11.745974+0000","last_fullsized":"2024-04-25T19:44:11.745974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:25.996441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435340+0000","last_change":"2024-04-25T19:44:10.067089+0000","last_active":"2024-04-25T19:44:11.435340+0000","last_peered":"2024-04-25T19:44:11.435340+0000","last_clean":"2024-04-25T19:44:11.435340+0000","last_became_active":"2024-04-25T19:44:10.065329+0000","last_became_peered":"2024-04-25T19:44:10.065329+0000","last_unstale":"2024-04-25T19:44:11.435340+0000","last_undegraded":"2024-04-25T19:44:11.435340+0000","last_fullsized":"2024-04-25T19:44:11.435340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:51.252629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436700+0000","last_change":"2024-04-25T19:44:10.060616+0000","last_active":"2024-04-25T19:44:11.436700+0000","last_peered":"2024-04-25T19:44:11.436700+0000","last_clean":"2024-04-25T19:44:11.436700+0000","last_became_active":"2024-04-25T19:44:10.060285+0000","last_became_peered":"2024-04-25T19:44:10.060285+0000","last_unstale":"2024-04-25T19:44:11.436700+0000","last_undegraded":"2024-04-25T19:44:11.436700+0000","last_fullsized":"2024-04-25T19:44:11.436700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:23:38.854656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435812+0000","last_change":"2024-04-25T19:44:09.059263+0000","last_active":"2024-04-25T19:44:11.435812+0000","last_peered":"2024-04-25T19:44:11.435812+0000","last_clean":"2024-04-25T19:44:11.435812+0000","last_became_active":"2024-04-25T19:44:09.058936+0000","last_became_peered":"2024-04-25T19:44:09.058936+0000","last_unstale":"2024-04-25T19:44:11.435812+0000","last_undegraded":"2024-04-25T19:44:11.435812+0000","last_fullsized":"2024-04-25T19:44:11.435812+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:49.562956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042882+0000","last_change":"2024-04-25T19:44:10.051152+0000","last_active":"2024-04-25T19:44:11.042882+0000","last_peered":"2024-04-25T19:44:11.042882+0000","last_clean":"2024-04-25T19:44:11.042882+0000","last_became_active":"2024-04-25T19:44:10.050607+0000","last_became_peered":"2024-04-25T19:44:10.050607+0000","last_unstale":"2024-04-25T19:44:11.042882+0000","last_undegraded":"2024-04-25T19:44:11.042882+0000","last_fullsized":"2024-04-25T19:44:11.042882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:28.992782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.744879+0000","last_change":"2024-04-25T19:44:09.054711+0000","last_active":"2024-04-25T19:44:11.744879+0000","last_peered":"2024-04-25T19:44:11.744879+0000","last_clean":"2024-04-25T19:44:11.744879+0000","last_became_active":"2024-04-25T19:44:09.053989+0000","last_became_peered":"2024-04-25T19:44:09.053989+0000","last_unstale":"2024-04-25T19:44:11.744879+0000","last_undegraded":"2024-04-25T19:44:11.744879+0000","last_fullsized":"2024-04-25T19:44:11.744879+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:25.699599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436322+0000","last_change":"2024-04-25T19:44:10.059956+0000","last_active":"2024-04-25T19:44:11.436322+0000","last_peered":"2024-04-25T19:44:11.436322+0000","last_clean":"2024-04-25T19:44:11.436322+0000","last_became_active":"2024-04-25T19:44:10.058700+0000","last_became_peered":"2024-04-25T19:44:10.058700+0000","last_unstale":"2024-04-25T19:44:11.436322+0000","last_undegraded":"2024-04-25T19:44:11.436322+0000","last_fullsized":"2024-04-25T19:44:11.436322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:25:09.001237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745309+0000","last_change":"2024-04-25T19:44:09.044537+0000","last_active":"2024-04-25T19:44:11.745309+0000","last_peered":"2024-04-25T19:44:11.745309+0000","last_clean":"2024-04-25T19:44:11.745309+0000","last_became_active":"2024-04-25T19:44:09.044205+0000","last_became_peered":"2024-04-25T19:44:09.044205+0000","last_unstale":"2024-04-25T19:44:11.745309+0000","last_undegraded":"2024-04-25T19:44:11.745309+0000","last_fullsized":"2024-04-25T19:44:11.745309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:11:22.034588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042844+0000","last_change":"2024-04-25T19:44:10.050365+0000","last_active":"2024-04-25T19:44:11.042844+0000","last_peered":"2024-04-25T19:44:11.042844+0000","last_clean":"2024-04-25T19:44:11.042844+0000","last_became_active":"2024-04-25T19:44:10.049730+0000","last_became_peered":"2024-04-25T19:44:10.049730+0000","last_unstale":"2024-04-25T19:44:11.042844+0000","last_undegraded":"2024-04-25T19:44:11.042844+0000","last_fullsized":"2024-04-25T19:44:11.042844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:26:17.991276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436186+0000","last_change":"2024-04-25T19:44:09.058547+0000","last_active":"2024-04-25T19:44:11.436186+0000","last_peered":"2024-04-25T19:44:11.436186+0000","last_clean":"2024-04-25T19:44:11.436186+0000","last_became_active":"2024-04-25T19:44:09.057348+0000","last_became_peered":"2024-04-25T19:44:09.057348+0000","last_unstale":"2024-04-25T19:44:11.436186+0000","last_undegraded":"2024-04-25T19:44:11.436186+0000","last_fullsized":"2024-04-25T19:44:11.436186+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:38:52.992542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042693+0000","last_change":"2024-04-25T19:44:10.047756+0000","last_active":"2024-04-25T19:44:11.042693+0000","last_peered":"2024-04-25T19:44:11.042693+0000","last_clean":"2024-04-25T19:44:11.042693+0000","last_became_active":"2024-04-25T19:44:10.047155+0000","last_became_peered":"2024-04-25T19:44:10.047155+0000","last_unstale":"2024-04-25T19:44:11.042693+0000","last_undegraded":"2024-04-25T19:44:11.042693+0000","last_fullsized":"2024-04-25T19:44:11.042693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:43:23.290058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745329+0000","last_change":"2024-04-25T19:44:09.053455+0000","last_active":"2024-04-25T19:44:11.745329+0000","last_peered":"2024-04-25T19:44:11.745329+0000","last_clean":"2024-04-25T19:44:11.745329+0000","last_became_active":"2024-04-25T19:44:09.053118+0000","last_became_peered":"2024-04-25T19:44:09.053118+0000","last_unstale":"2024-04-25T19:44:11.745329+0000","last_undegraded":"2024-04-25T19:44:11.745329+0000","last_fullsized":"2024-04-25T19:44:11.745329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:59:23.033455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043396+0000","last_change":"2024-04-25T19:44:10.047937+0000","last_active":"2024-04-25T19:44:11.043396+0000","last_peered":"2024-04-25T19:44:11.043396+0000","last_clean":"2024-04-25T19:44:11.043396+0000","last_became_active":"2024-04-25T19:44:10.046231+0000","last_became_peered":"2024-04-25T19:44:10.046231+0000","last_unstale":"2024-04-25T19:44:11.043396+0000","last_undegraded":"2024-04-25T19:44:11.043396+0000","last_fullsized":"2024-04-25T19:44:11.043396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:50.564937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436779+0000","last_change":"2024-04-25T19:44:09.060536+0000","last_active":"2024-04-25T19:44:11.436779+0000","last_peered":"2024-04-25T19:44:11.436779+0000","last_clean":"2024-04-25T19:44:11.436779+0000","last_became_active":"2024-04-25T19:44:09.059473+0000","last_became_peered":"2024-04-25T19:44:09.059473+0000","last_unstale":"2024-04-25T19:44:11.436779+0000","last_undegraded":"2024-04-25T19:44:11.436779+0000","last_fullsized":"2024-04-25T19:44:11.436779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:20.907667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435968+0000","last_change":"2024-04-25T19:44:10.043795+0000","last_active":"2024-04-25T19:44:11.435968+0000","last_peered":"2024-04-25T19:44:11.435968+0000","last_clean":"2024-04-25T19:44:11.435968+0000","last_became_active":"2024-04-25T19:44:10.043342+0000","last_became_peered":"2024-04-25T19:44:10.043342+0000","last_unstale":"2024-04-25T19:44:11.435968+0000","last_undegraded":"2024-04-25T19:44:11.435968+0000","last_fullsized":"2024-04-25T19:44:11.435968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:52:49.026097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042861+0000","last_change":"2024-04-25T19:44:09.044583+0000","last_active":"2024-04-25T19:44:11.042861+0000","last_peered":"2024-04-25T19:44:11.042861+0000","last_clean":"2024-04-25T19:44:11.042861+0000","last_became_active":"2024-04-25T19:44:09.044225+0000","last_became_peered":"2024-04-25T19:44:09.044225+0000","last_unstale":"2024-04-25T19:44:11.042861+0000","last_undegraded":"2024-04-25T19:44:11.042861+0000","last_fullsized":"2024-04-25T19:44:11.042861+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:25.699599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042456+0000","last_change":"2024-04-25T19:44:10.049048+0000","last_active":"2024-04-25T19:44:11.042456+0000","last_peered":"2024-04-25T19:44:11.042456+0000","last_clean":"2024-04-25T19:44:11.042456+0000","last_became_active":"2024-04-25T19:44:10.048740+0000","last_became_peered":"2024-04-25T19:44:10.048740+0000","last_unstale":"2024-04-25T19:44:11.042456+0000","last_undegraded":"2024-04-25T19:44:11.042456+0000","last_fullsized":"2024-04-25T19:44:11.042456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:19.654106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745972+0000","last_change":"2024-04-25T19:44:09.046229+0000","last_active":"2024-04-25T19:44:11.745972+0000","last_peered":"2024-04-25T19:44:11.745972+0000","last_clean":"2024-04-25T19:44:11.745972+0000","last_became_active":"2024-04-25T19:44:09.045423+0000","last_became_peered":"2024-04-25T19:44:09.045423+0000","last_unstale":"2024-04-25T19:44:11.745972+0000","last_undegraded":"2024-04-25T19:44:11.745972+0000","last_fullsized":"2024-04-25T19:44:11.745972+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:30:43.807694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672967,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":64550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672967,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":654870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672967,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":790528,"data_stored":653786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":188416,"data_stored":63326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T19:44:18.068 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T19:44:18.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.214+0000 7f5c5a71b640 1 Processor -- start 2024-04-25T19:44:18.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.214+0000 7f5c5a71b640 1 -- start start 2024-04-25T19:44:18.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.215+0000 7f5c5a71b640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5c541438d0 0x7f5c54143ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:18.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.215+0000 7f5c5a71b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5c5414c1a0 0x7f5c5414e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:18.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.215+0000 7f5c5a71b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c540a28e0 con 0x7f5c540a1ef0 2024-04-25T19:44:18.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.215+0000 7f5c5a71b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f5c541443a0 con 0x7f5c5414c1a0 2024-04-25T19:44:18.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.215+0000 7f5c5a71b640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f5c54144660 con 0x7f5c541438d0 2024-04-25T19:44:18.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.215+0000 7f5c58cb8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5c5414c1a0 0x7f5c5414e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:18.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.215+0000 7f5c53fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5c540a1ef0 0x7f5c540a5d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52916/0 (socket says 172.21.15.29:52916) 2024-04-25T19:44:18.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.215+0000 7f5c53fff640 1 -- 172.21.15.29:0/1122524167 learned_addr learned my addr 172.21.15.29:0/1122524167 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:18.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.215+0000 7f5c537fe640 1 --2- 172.21.15.29:0/1122524167 >> v2:172.21.15.102:3300/0 conn(0x7f5c541438d0 0x7f5c54143ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:18.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.215+0000 7f5c58cb8640 1 -- 172.21.15.29:0/1122524167 >> v2:172.21.15.102:3300/0 conn(0x7f5c541438d0 msgr2=0x7f5c54143ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:18.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.215+0000 7f5c58cb8640 1 --2- 172.21.15.29:0/1122524167 >> v2:172.21.15.102:3300/0 conn(0x7f5c541438d0 0x7f5c54143ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.215+0000 7f5c58cb8640 1 -- 172.21.15.29:0/1122524167 >> v1:172.21.15.29:6789/0 conn(0x7f5c540a1ef0 legacy=0x7f5c540a5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:18.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.215+0000 7f5c52ffd640 1 -- 172.21.15.29:0/1122524167 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3771397026 0 0) 0x7f5c540a28e0 con 0x7f5c540a1ef0 2024-04-25T19:44:18.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.215+0000 7f5c58cb8640 1 -- 172.21.15.29:0/1122524167 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c540a1710 con 0x7f5c5414c1a0 2024-04-25T19:44:18.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.216+0000 7f5c537fe640 1 --2- 172.21.15.29:0/1122524167 >> v2:172.21.15.102:3300/0 conn(0x7f5c541438d0 0x7f5c54143ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:44:18.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.216+0000 7f5c58cb8640 1 --2- 172.21.15.29:0/1122524167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c5414c1a0 0x7f5c5414e5c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5c5406f5d0 tx=0x7f5c4c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3986a50e878f3d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:18.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.216+0000 7f5c52ffd640 1 -- 172.21.15.29:0/1122524167 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c039070 con 0x7f5c5414c1a0 2024-04-25T19:44:18.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.216+0000 7f5c52ffd640 1 -- 172.21.15.29:0/1122524167 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c4c02cc80 con 0x7f5c5414c1a0 2024-04-25T19:44:18.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.216+0000 7f5c52ffd640 1 -- 172.21.15.29:0/1122524167 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c034660 con 0x7f5c5414c1a0 2024-04-25T19:44:18.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.217+0000 7f5c5a71b640 1 -- 172.21.15.29:0/1122524167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c5414c1a0 msgr2=0x7f5c5414e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:18.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.217+0000 7f5c5a71b640 1 --2- 172.21.15.29:0/1122524167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c5414c1a0 0x7f5c5414e5c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5c5406f5d0 tx=0x7f5c4c02bed0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.217+0000 7f5c5a71b640 1 -- 172.21.15.29:0/1122524167 shutdown_connections 2024-04-25T19:44:18.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.217+0000 7f5c5a71b640 1 --2- 172.21.15.29:0/1122524167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c5414c1a0 0x7f5c5414e5c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.217+0000 7f5c5a71b640 1 --2- 172.21.15.29:0/1122524167 >> v2:172.21.15.102:3300/0 conn(0x7f5c541438d0 0x7f5c54143ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.217+0000 7f5c5a71b640 1 -- 172.21.15.29:0/1122524167 >> 172.21.15.29:0/1122524167 conn(0x7f5c540a6f10 msgr2=0x7f5c540a7320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:18.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.217+0000 7f5c5a71b640 1 -- 172.21.15.29:0/1122524167 shutdown_connections 2024-04-25T19:44:18.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.217+0000 7f5c5a71b640 1 -- 172.21.15.29:0/1122524167 wait complete. 2024-04-25T19:44:18.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.219+0000 7f5c5a71b640 1 Processor -- start 2024-04-25T19:44:18.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.219+0000 7f5c5a71b640 1 -- start start 2024-04-25T19:44:18.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.220+0000 7f5c5a71b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540a1ef0 0x7f5c540a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:18.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.220+0000 7f5c5a71b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5c541438d0 0x7f5c5409af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:18.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.220+0000 7f5c5a71b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c5414c1a0 0x7f5c541daa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:18.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.220+0000 7f5c5a71b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5c540a2480 con 0x7f5c540a1ef0 2024-04-25T19:44:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.220+0000 7f5c5a71b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5c540a2300 con 0x7f5c5414c1a0 2024-04-25T19:44:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.220+0000 7f5c5a71b640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f5c540a1310 con 0x7f5c541438d0 2024-04-25T19:44:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.220+0000 7f5c53fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540a1ef0 0x7f5c540a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.220+0000 7f5c537fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5c541438d0 0x7f5c5409af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.220+0000 7f5c53fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540a1ef0 0x7f5c540a5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39736/0 (socket says 172.21.15.29:39736) 2024-04-25T19:44:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.220+0000 7f5c53fff640 1 -- 172.21.15.29:0/787129308 learned_addr learned my addr 172.21.15.29:0/787129308 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.220+0000 7f5c58cb8640 1 --2- 172.21.15.29:0/787129308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c5414c1a0 0x7f5c541daa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.220+0000 7f5c537fe640 1 -- 172.21.15.29:0/787129308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c5414c1a0 msgr2=0x7f5c541daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.220+0000 7f5c537fe640 1 --2- 172.21.15.29:0/787129308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c5414c1a0 0x7f5c541daa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.220+0000 7f5c537fe640 1 -- 172.21.15.29:0/787129308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540a1ef0 msgr2=0x7f5c540a5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.220+0000 7f5c537fe640 1 --2- 172.21.15.29:0/787129308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540a1ef0 0x7f5c540a5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.220+0000 7f5c537fe640 1 -- 172.21.15.29:0/787129308 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c5409b470 con 0x7f5c541438d0 2024-04-25T19:44:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.220+0000 7f5c53fff640 1 --2- 172.21.15.29:0/787129308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540a1ef0 0x7f5c540a5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.220+0000 7f5c537fe640 1 --2- 172.21.15.29:0/787129308 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5c541438d0 0x7f5c5409af30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5c4400d7d0 tx=0x7f5c4400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.221+0000 7f5c50ff9640 1 -- 172.21.15.29:0/787129308 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c44004030 con 0x7f5c541438d0 2024-04-25T19:44:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.221+0000 7f5c50ff9640 1 -- 172.21.15.29:0/787129308 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c440041d0 con 0x7f5c541438d0 2024-04-25T19:44:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.221+0000 7f5c5a71b640 1 -- 172.21.15.29:0/787129308 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c541da570 con 0x7f5c541438d0 2024-04-25T19:44:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.221+0000 7f5c50ff9640 1 -- 172.21.15.29:0/787129308 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4400b6c0 con 0x7f5c541438d0 2024-04-25T19:44:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.221+0000 7f5c5a71b640 1 -- 172.21.15.29:0/787129308 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c541dafd0 con 0x7f5c541438d0 2024-04-25T19:44:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.222+0000 7f5c5a71b640 1 -- 172.21.15.29:0/787129308 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c541db2f0 con 0x7f5c541438d0 2024-04-25T19:44:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.223+0000 7f5c50ff9640 1 -- 172.21.15.29:0/787129308 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f5c4400b860 con 0x7f5c541438d0 2024-04-25T19:44:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.223+0000 7f5c50ff9640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.224+0000 7f5c50ff9640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.224+0000 7f5c50ff9640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.224+0000 7f5c50ff9640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.224+0000 7f5c50ff9640 1 --2- 172.21.15.29:0/787129308 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5c400ac4b0 0x7f5c400ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.224+0000 7f5c50ff9640 1 -- 172.21.15.29:0/787129308 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f5c440da110 con 0x7f5c541438d0 2024-04-25T19:44:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.224+0000 7f5c53fff640 1 --2- 172.21.15.29:0/787129308 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5c400ac4b0 0x7f5c400ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:18.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.225+0000 7f5c53fff640 1 --2- 172.21.15.29:0/787129308 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5c400ac4b0 0x7f5c400ae9c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5c48000f60 tx=0x7f5c48007ea0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:18.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.228+0000 7f5c50ff9640 1 -- 172.21.15.29:0/787129308 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5c541db2f0 con 0x7f5c541438d0 2024-04-25T19:44:18.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.409+0000 7f5c5a71b640 20 mgrc start_command cmd: [{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}] 2024-04-25T19:44:18.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.409+0000 7f5c5a71b640 1 -- 172.21.15.29:0/787129308 --> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5c540507b0 con 0x7f5c400ac4b0 2024-04-25T19:44:18.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.416+0000 7f5c50ff9640 1 -- 172.21.15.29:0/787129308 <== mgr.4108 v2:172.21.15.102:6824/1439704316 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+268533 (secure 0 0 0) 0x7f5c540507b0 con 0x7f5c400ac4b0 2024-04-25T19:44:18.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.416+0000 7f5c50ff9640 20 mgrc handle_command_reply tid 0 r 0 2024-04-25T19:44:18.421 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:18.424 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-04-25T19:44:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.419+0000 7f5c5a71b640 10 mgrc shutdown 2024-04-25T19:44:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.419+0000 7f5c5a71b640 1 -- 172.21.15.29:0/787129308 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5c400ac4b0 msgr2=0x7f5c400ae9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.419+0000 7f5c5a71b640 1 --2- 172.21.15.29:0/787129308 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5c400ac4b0 0x7f5c400ae9c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5c48000f60 tx=0x7f5c48007ea0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.420+0000 7f5c5a71b640 1 -- 172.21.15.29:0/787129308 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5c541438d0 msgr2=0x7f5c5409af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.420+0000 7f5c5a71b640 1 --2- 172.21.15.29:0/787129308 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5c541438d0 0x7f5c5409af30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5c4400d7d0 tx=0x7f5c4400dc90 comp rx=0 tx=0).stop 2024-04-25T19:44:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.420+0000 7f5c5a71b640 1 -- 172.21.15.29:0/787129308 shutdown_connections 2024-04-25T19:44:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.420+0000 7f5c5a71b640 1 --2- 172.21.15.29:0/787129308 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5c400ac4b0 0x7f5c400ae9c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.420+0000 7f5c5a71b640 1 --2- 172.21.15.29:0/787129308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c5414c1a0 0x7f5c541daa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.420+0000 7f5c5a71b640 1 --2- 172.21.15.29:0/787129308 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5c541438d0 0x7f5c5409af30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.420+0000 7f5c5a71b640 1 --2- 172.21.15.29:0/787129308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c540a1ef0 0x7f5c540a5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.420+0000 7f5c5a71b640 1 -- 172.21.15.29:0/787129308 >> 172.21.15.29:0/787129308 conn(0x7f5c540a6f10 msgr2=0x7f5c5406e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:18.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.420+0000 7f5c5a71b640 1 -- 172.21.15.29:0/787129308 shutdown_connections 2024-04-25T19:44:18.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.420+0000 7f5c5a71b640 1 -- 172.21.15.29:0/787129308 wait complete. 2024-04-25T19:44:18.432 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-04-25T19:44:18.084354+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":114192,"kb_used_data":1944,"kb_used_omap":6,"kb_used_meta":112185,"kb_avail":377373168,"statfs":{"total":386547056640,"available":386430124032,"internally_reserved":0,"allocated":1990656,"data_stored":1436532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":114878230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.055989"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043136+0000","last_change":"2024-04-25T19:44:10.065844+0000","last_active":"2024-04-25T19:44:11.043136+0000","last_peered":"2024-04-25T19:44:11.043136+0000","last_clean":"2024-04-25T19:44:11.043136+0000","last_became_active":"2024-04-25T19:44:10.065543+0000","last_became_peered":"2024-04-25T19:44:10.065543+0000","last_unstale":"2024-04-25T19:44:11.043136+0000","last_undegraded":"2024-04-25T19:44:11.043136+0000","last_fullsized":"2024-04-25T19:44:11.043136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:28.992782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042952+0000","last_change":"2024-04-25T19:44:10.054974+0000","last_active":"2024-04-25T19:44:11.042952+0000","last_peered":"2024-04-25T19:44:11.042952+0000","last_clean":"2024-04-25T19:44:11.042952+0000","last_became_active":"2024-04-25T19:44:10.054042+0000","last_became_peered":"2024-04-25T19:44:10.054042+0000","last_unstale":"2024-04-25T19:44:11.042952+0000","last_undegraded":"2024-04-25T19:44:11.042952+0000","last_fullsized":"2024-04-25T19:44:11.042952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:56.154645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436639+0000","last_change":"2024-04-25T19:44:10.060234+0000","last_active":"2024-04-25T19:44:11.436639+0000","last_peered":"2024-04-25T19:44:11.436639+0000","last_clean":"2024-04-25T19:44:11.436639+0000","last_became_active":"2024-04-25T19:44:10.059094+0000","last_became_peered":"2024-04-25T19:44:10.059094+0000","last_unstale":"2024-04-25T19:44:11.436639+0000","last_undegraded":"2024-04-25T19:44:11.436639+0000","last_fullsized":"2024-04-25T19:44:11.436639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:18.383491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745013+0000","last_change":"2024-04-25T19:44:10.056978+0000","last_active":"2024-04-25T19:44:11.745013+0000","last_peered":"2024-04-25T19:44:11.745013+0000","last_clean":"2024-04-25T19:44:11.745013+0000","last_became_active":"2024-04-25T19:44:10.051835+0000","last_became_peered":"2024-04-25T19:44:10.051835+0000","last_unstale":"2024-04-25T19:44:11.745013+0000","last_undegraded":"2024-04-25T19:44:11.745013+0000","last_fullsized":"2024-04-25T19:44:11.745013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:05:36.546123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436228+0000","last_change":"2024-04-25T19:44:10.058920+0000","last_active":"2024-04-25T19:44:11.436228+0000","last_peered":"2024-04-25T19:44:11.436228+0000","last_clean":"2024-04-25T19:44:11.436228+0000","last_became_active":"2024-04-25T19:44:10.058585+0000","last_became_peered":"2024-04-25T19:44:10.058585+0000","last_unstale":"2024-04-25T19:44:11.436228+0000","last_undegraded":"2024-04-25T19:44:11.436228+0000","last_fullsized":"2024-04-25T19:44:11.436228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:48:06.055133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435841+0000","last_change":"2024-04-25T19:44:10.055592+0000","last_active":"2024-04-25T19:44:11.435841+0000","last_peered":"2024-04-25T19:44:11.435841+0000","last_clean":"2024-04-25T19:44:11.435841+0000","last_became_active":"2024-04-25T19:44:10.053926+0000","last_became_peered":"2024-04-25T19:44:10.053926+0000","last_unstale":"2024-04-25T19:44:11.435841+0000","last_undegraded":"2024-04-25T19:44:11.435841+0000","last_fullsized":"2024-04-25T19:44:11.435841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:01.889093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435699+0000","last_change":"2024-04-25T19:44:10.058739+0000","last_active":"2024-04-25T19:44:11.435699+0000","last_peered":"2024-04-25T19:44:11.435699+0000","last_clean":"2024-04-25T19:44:11.435699+0000","last_became_active":"2024-04-25T19:44:10.053725+0000","last_became_peered":"2024-04-25T19:44:10.053725+0000","last_unstale":"2024-04-25T19:44:11.435699+0000","last_undegraded":"2024-04-25T19:44:11.435699+0000","last_fullsized":"2024-04-25T19:44:11.435699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:49.082468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745508+0000","last_change":"2024-04-25T19:44:10.069350+0000","last_active":"2024-04-25T19:44:11.745508+0000","last_peered":"2024-04-25T19:44:11.745508+0000","last_clean":"2024-04-25T19:44:11.745508+0000","last_became_active":"2024-04-25T19:44:10.069065+0000","last_became_peered":"2024-04-25T19:44:10.069065+0000","last_unstale":"2024-04-25T19:44:11.745508+0000","last_undegraded":"2024-04-25T19:44:11.745508+0000","last_fullsized":"2024-04-25T19:44:11.745508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:43:06.731936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042767+0000","last_change":"2024-04-25T19:44:10.052909+0000","last_active":"2024-04-25T19:44:11.042767+0000","last_peered":"2024-04-25T19:44:11.042767+0000","last_clean":"2024-04-25T19:44:11.042767+0000","last_became_active":"2024-04-25T19:44:10.052592+0000","last_became_peered":"2024-04-25T19:44:10.052592+0000","last_unstale":"2024-04-25T19:44:11.042767+0000","last_undegraded":"2024-04-25T19:44:11.042767+0000","last_fullsized":"2024-04-25T19:44:11.042767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:50.564937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745396+0000","last_change":"2024-04-25T19:44:10.060703+0000","last_active":"2024-04-25T19:44:11.745396+0000","last_peered":"2024-04-25T19:44:11.745396+0000","last_clean":"2024-04-25T19:44:11.745396+0000","last_became_active":"2024-04-25T19:44:10.058013+0000","last_became_peered":"2024-04-25T19:44:10.058013+0000","last_unstale":"2024-04-25T19:44:11.745396+0000","last_undegraded":"2024-04-25T19:44:11.745396+0000","last_fullsized":"2024-04-25T19:44:11.745396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:15:17.871931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745517+0000","last_change":"2024-04-25T19:44:10.049572+0000","last_active":"2024-04-25T19:44:11.745517+0000","last_peered":"2024-04-25T19:44:11.745517+0000","last_clean":"2024-04-25T19:44:11.745517+0000","last_became_active":"2024-04-25T19:44:10.049249+0000","last_became_peered":"2024-04-25T19:44:10.049249+0000","last_unstale":"2024-04-25T19:44:11.745517+0000","last_undegraded":"2024-04-25T19:44:11.745517+0000","last_fullsized":"2024-04-25T19:44:11.745517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:52:09.103201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043197+0000","last_change":"2024-04-25T19:44:10.056274+0000","last_active":"2024-04-25T19:44:11.043197+0000","last_peered":"2024-04-25T19:44:11.043197+0000","last_clean":"2024-04-25T19:44:11.043197+0000","last_became_active":"2024-04-25T19:44:10.055981+0000","last_became_peered":"2024-04-25T19:44:10.055981+0000","last_unstale":"2024-04-25T19:44:11.043197+0000","last_undegraded":"2024-04-25T19:44:11.043197+0000","last_fullsized":"2024-04-25T19:44:11.043197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:52:52.432869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042230+0000","last_change":"2024-04-25T19:44:10.044363+0000","last_active":"2024-04-25T19:44:11.042230+0000","last_peered":"2024-04-25T19:44:11.042230+0000","last_clean":"2024-04-25T19:44:11.042230+0000","last_became_active":"2024-04-25T19:44:10.044033+0000","last_became_peered":"2024-04-25T19:44:10.044033+0000","last_unstale":"2024-04-25T19:44:11.042230+0000","last_undegraded":"2024-04-25T19:44:11.042230+0000","last_fullsized":"2024-04-25T19:44:11.042230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:19.654106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.744782+0000","last_change":"2024-04-25T19:44:10.068850+0000","last_active":"2024-04-25T19:44:11.744782+0000","last_peered":"2024-04-25T19:44:11.744782+0000","last_clean":"2024-04-25T19:44:11.744782+0000","last_became_active":"2024-04-25T19:44:10.068565+0000","last_became_peered":"2024-04-25T19:44:10.068565+0000","last_unstale":"2024-04-25T19:44:11.744782+0000","last_undegraded":"2024-04-25T19:44:11.744782+0000","last_fullsized":"2024-04-25T19:44:11.744782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:40.201502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.744781+0000","last_change":"2024-04-25T19:44:10.056683+0000","last_active":"2024-04-25T19:44:11.744781+0000","last_peered":"2024-04-25T19:44:11.744781+0000","last_clean":"2024-04-25T19:44:11.744781+0000","last_became_active":"2024-04-25T19:44:10.056375+0000","last_became_peered":"2024-04-25T19:44:10.056375+0000","last_unstale":"2024-04-25T19:44:11.744781+0000","last_undegraded":"2024-04-25T19:44:11.744781+0000","last_fullsized":"2024-04-25T19:44:11.744781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:50:21.528308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435257+0000","last_change":"2024-04-25T19:44:10.056384+0000","last_active":"2024-04-25T19:44:11.435257+0000","last_peered":"2024-04-25T19:44:11.435257+0000","last_clean":"2024-04-25T19:44:11.435257+0000","last_became_active":"2024-04-25T19:44:10.054724+0000","last_became_peered":"2024-04-25T19:44:10.054724+0000","last_unstale":"2024-04-25T19:44:11.435257+0000","last_undegraded":"2024-04-25T19:44:11.435257+0000","last_fullsized":"2024-04-25T19:44:11.435257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:46:59.212591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745764+0000","last_change":"2024-04-25T19:44:10.069348+0000","last_active":"2024-04-25T19:44:11.745764+0000","last_peered":"2024-04-25T19:44:11.745764+0000","last_clean":"2024-04-25T19:44:11.745764+0000","last_became_active":"2024-04-25T19:44:10.068699+0000","last_became_peered":"2024-04-25T19:44:10.068699+0000","last_unstale":"2024-04-25T19:44:11.745764+0000","last_undegraded":"2024-04-25T19:44:11.745764+0000","last_fullsized":"2024-04-25T19:44:11.745764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:33:58.495931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436501+0000","last_change":"2024-04-25T19:44:10.061949+0000","last_active":"2024-04-25T19:44:11.436501+0000","last_peered":"2024-04-25T19:44:11.436501+0000","last_clean":"2024-04-25T19:44:11.436501+0000","last_became_active":"2024-04-25T19:44:10.061187+0000","last_became_peered":"2024-04-25T19:44:10.061187+0000","last_unstale":"2024-04-25T19:44:11.436501+0000","last_undegraded":"2024-04-25T19:44:11.436501+0000","last_fullsized":"2024-04-25T19:44:11.436501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:41:07.670796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042266+0000","last_change":"2024-04-25T19:44:10.045506+0000","last_active":"2024-04-25T19:44:11.042266+0000","last_peered":"2024-04-25T19:44:11.042266+0000","last_clean":"2024-04-25T19:44:11.042266+0000","last_became_active":"2024-04-25T19:44:10.044359+0000","last_became_peered":"2024-04-25T19:44:10.044359+0000","last_unstale":"2024-04-25T19:44:11.042266+0000","last_undegraded":"2024-04-25T19:44:11.042266+0000","last_fullsized":"2024-04-25T19:44:11.042266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:43:23.290058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745208+0000","last_change":"2024-04-25T19:44:10.051944+0000","last_active":"2024-04-25T19:44:11.745208+0000","last_peered":"2024-04-25T19:44:11.745208+0000","last_clean":"2024-04-25T19:44:11.745208+0000","last_became_active":"2024-04-25T19:44:10.051469+0000","last_became_peered":"2024-04-25T19:44:10.051469+0000","last_unstale":"2024-04-25T19:44:11.745208+0000","last_undegraded":"2024-04-25T19:44:11.745208+0000","last_fullsized":"2024-04-25T19:44:11.745208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:46:54.526883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745057+0000","last_change":"2024-04-25T19:44:10.060425+0000","last_active":"2024-04-25T19:44:11.745057+0000","last_peered":"2024-04-25T19:44:11.745057+0000","last_clean":"2024-04-25T19:44:11.745057+0000","last_became_active":"2024-04-25T19:44:10.059664+0000","last_became_peered":"2024-04-25T19:44:10.059664+0000","last_unstale":"2024-04-25T19:44:11.745057+0000","last_undegraded":"2024-04-25T19:44:11.745057+0000","last_fullsized":"2024-04-25T19:44:11.745057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:10:11.581750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.744779+0000","last_change":"2024-04-25T19:44:10.068792+0000","last_active":"2024-04-25T19:44:11.744779+0000","last_peered":"2024-04-25T19:44:11.744779+0000","last_clean":"2024-04-25T19:44:11.744779+0000","last_became_active":"2024-04-25T19:44:10.068477+0000","last_became_peered":"2024-04-25T19:44:10.068477+0000","last_unstale":"2024-04-25T19:44:11.744779+0000","last_undegraded":"2024-04-25T19:44:11.744779+0000","last_fullsized":"2024-04-25T19:44:11.744779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:18.399916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435517+0000","last_change":"2024-04-25T19:44:10.058966+0000","last_active":"2024-04-25T19:44:11.435517+0000","last_peered":"2024-04-25T19:44:11.435517+0000","last_clean":"2024-04-25T19:44:11.435517+0000","last_became_active":"2024-04-25T19:44:10.058083+0000","last_became_peered":"2024-04-25T19:44:10.058083+0000","last_unstale":"2024-04-25T19:44:11.435517+0000","last_undegraded":"2024-04-25T19:44:11.435517+0000","last_fullsized":"2024-04-25T19:44:11.435517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:34.810052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745205+0000","last_change":"2024-04-25T19:44:10.057670+0000","last_active":"2024-04-25T19:44:11.745205+0000","last_peered":"2024-04-25T19:44:11.745205+0000","last_clean":"2024-04-25T19:44:11.745205+0000","last_became_active":"2024-04-25T19:44:10.057360+0000","last_became_peered":"2024-04-25T19:44:10.057360+0000","last_unstale":"2024-04-25T19:44:11.745205+0000","last_undegraded":"2024-04-25T19:44:11.745205+0000","last_fullsized":"2024-04-25T19:44:11.745205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:18:02.918434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745808+0000","last_change":"2024-04-25T19:44:10.069281+0000","last_active":"2024-04-25T19:44:11.745808+0000","last_peered":"2024-04-25T19:44:11.745808+0000","last_clean":"2024-04-25T19:44:11.745808+0000","last_became_active":"2024-04-25T19:44:10.068327+0000","last_became_peered":"2024-04-25T19:44:10.068327+0000","last_unstale":"2024-04-25T19:44:11.745808+0000","last_undegraded":"2024-04-25T19:44:11.745808+0000","last_fullsized":"2024-04-25T19:44:11.745808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:46:59.212591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042716+0000","last_change":"2024-04-25T19:44:10.055433+0000","last_active":"2024-04-25T19:44:11.042716+0000","last_peered":"2024-04-25T19:44:11.042716+0000","last_clean":"2024-04-25T19:44:11.042716+0000","last_became_active":"2024-04-25T19:44:10.055058+0000","last_became_peered":"2024-04-25T19:44:10.055058+0000","last_unstale":"2024-04-25T19:44:11.042716+0000","last_undegraded":"2024-04-25T19:44:11.042716+0000","last_fullsized":"2024-04-25T19:44:11.042716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:56.154645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.437038+0000","last_change":"2024-04-25T19:44:10.064610+0000","last_active":"2024-04-25T19:44:11.437038+0000","last_peered":"2024-04-25T19:44:11.437038+0000","last_clean":"2024-04-25T19:44:11.437038+0000","last_became_active":"2024-04-25T19:44:10.064295+0000","last_became_peered":"2024-04-25T19:44:10.064295+0000","last_unstale":"2024-04-25T19:44:11.437038+0000","last_undegraded":"2024-04-25T19:44:11.437038+0000","last_fullsized":"2024-04-25T19:44:11.437038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:09.492760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043265+0000","last_change":"2024-04-25T19:44:10.050485+0000","last_active":"2024-04-25T19:44:11.043265+0000","last_peered":"2024-04-25T19:44:11.043265+0000","last_clean":"2024-04-25T19:44:11.043265+0000","last_became_active":"2024-04-25T19:44:10.050141+0000","last_became_peered":"2024-04-25T19:44:10.050141+0000","last_unstale":"2024-04-25T19:44:11.043265+0000","last_undegraded":"2024-04-25T19:44:11.043265+0000","last_fullsized":"2024-04-25T19:44:11.043265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:40.201502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042339+0000","last_change":"2024-04-25T19:44:10.044738+0000","last_active":"2024-04-25T19:44:11.042339+0000","last_peered":"2024-04-25T19:44:11.042339+0000","last_clean":"2024-04-25T19:44:11.042339+0000","last_became_active":"2024-04-25T19:44:10.044326+0000","last_became_peered":"2024-04-25T19:44:10.044326+0000","last_unstale":"2024-04-25T19:44:11.042339+0000","last_undegraded":"2024-04-25T19:44:11.042339+0000","last_fullsized":"2024-04-25T19:44:11.042339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:21.909648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745442+0000","last_change":"2024-04-25T19:44:10.062939+0000","last_active":"2024-04-25T19:44:11.745442+0000","last_peered":"2024-04-25T19:44:11.745442+0000","last_clean":"2024-04-25T19:44:11.745442+0000","last_became_active":"2024-04-25T19:44:10.057793+0000","last_became_peered":"2024-04-25T19:44:10.057793+0000","last_unstale":"2024-04-25T19:44:11.745442+0000","last_undegraded":"2024-04-25T19:44:11.745442+0000","last_fullsized":"2024-04-25T19:44:11.745442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:30.311117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.041761+0000","last_change":"2024-04-25T19:44:10.053059+0000","last_active":"2024-04-25T19:44:11.041761+0000","last_peered":"2024-04-25T19:44:11.041761+0000","last_clean":"2024-04-25T19:44:11.041761+0000","last_became_active":"2024-04-25T19:44:10.052494+0000","last_became_peered":"2024-04-25T19:44:10.052494+0000","last_unstale":"2024-04-25T19:44:11.041761+0000","last_undegraded":"2024-04-25T19:44:11.041761+0000","last_fullsized":"2024-04-25T19:44:11.041761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:38:53.994523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745521+0000","last_change":"2024-04-25T19:44:10.060010+0000","last_active":"2024-04-25T19:44:11.745521+0000","last_peered":"2024-04-25T19:44:11.745521+0000","last_clean":"2024-04-25T19:44:11.745521+0000","last_became_active":"2024-04-25T19:44:10.059598+0000","last_became_peered":"2024-04-25T19:44:10.059598+0000","last_unstale":"2024-04-25T19:44:11.745521+0000","last_undegraded":"2024-04-25T19:44:11.745521+0000","last_fullsized":"2024-04-25T19:44:11.745521+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:40.247386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435938+0000","last_change":"2024-04-25T19:44:10.059147+0000","last_active":"2024-04-25T19:44:11.435938+0000","last_peered":"2024-04-25T19:44:11.435938+0000","last_clean":"2024-04-25T19:44:11.435938+0000","last_became_active":"2024-04-25T19:44:10.058363+0000","last_became_peered":"2024-04-25T19:44:10.058363+0000","last_unstale":"2024-04-25T19:44:11.435938+0000","last_undegraded":"2024-04-25T19:44:11.435938+0000","last_fullsized":"2024-04-25T19:44:11.435938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:05:57.081479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436415+0000","last_change":"2024-04-25T19:44:09.056791+0000","last_active":"2024-04-25T19:44:11.436415+0000","last_peered":"2024-04-25T19:44:11.436415+0000","last_clean":"2024-04-25T19:44:11.436415+0000","last_became_active":"2024-04-25T19:44:09.056430+0000","last_became_peered":"2024-04-25T19:44:09.056430+0000","last_unstale":"2024-04-25T19:44:11.436415+0000","last_undegraded":"2024-04-25T19:44:11.436415+0000","last_fullsized":"2024-04-25T19:44:11.436415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:18.652125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436777+0000","last_change":"2024-04-25T19:44:10.062414+0000","last_active":"2024-04-25T19:44:11.436777+0000","last_peered":"2024-04-25T19:44:11.436777+0000","last_clean":"2024-04-25T19:44:11.436777+0000","last_became_active":"2024-04-25T19:44:10.061691+0000","last_became_peered":"2024-04-25T19:44:10.061691+0000","last_unstale":"2024-04-25T19:44:11.436777+0000","last_undegraded":"2024-04-25T19:44:11.436777+0000","last_fullsized":"2024-04-25T19:44:11.436777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:29:27.722435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043208+0000","last_change":"2024-04-25T19:44:09.052305+0000","last_active":"2024-04-25T19:44:11.043208+0000","last_peered":"2024-04-25T19:44:11.043208+0000","last_clean":"2024-04-25T19:44:11.043208+0000","last_became_active":"2024-04-25T19:44:09.051968+0000","last_became_peered":"2024-04-25T19:44:09.051968+0000","last_unstale":"2024-04-25T19:44:11.043208+0000","last_undegraded":"2024-04-25T19:44:11.043208+0000","last_fullsized":"2024-04-25T19:44:11.043208+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:11:22.034588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436495+0000","last_change":"2024-04-25T19:44:10.062199+0000","last_active":"2024-04-25T19:44:11.436495+0000","last_peered":"2024-04-25T19:44:11.436495+0000","last_clean":"2024-04-25T19:44:11.436495+0000","last_became_active":"2024-04-25T19:44:10.061613+0000","last_became_peered":"2024-04-25T19:44:10.061613+0000","last_unstale":"2024-04-25T19:44:11.436495+0000","last_undegraded":"2024-04-25T19:44:11.436495+0000","last_fullsized":"2024-04-25T19:44:11.436495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:56:49.470340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436342+0000","last_change":"2024-04-25T19:44:09.058706+0000","last_active":"2024-04-25T19:44:11.436342+0000","last_peered":"2024-04-25T19:44:11.436342+0000","last_clean":"2024-04-25T19:44:11.436342+0000","last_became_active":"2024-04-25T19:44:09.057009+0000","last_became_peered":"2024-04-25T19:44:09.057009+0000","last_unstale":"2024-04-25T19:44:11.436342+0000","last_undegraded":"2024-04-25T19:44:11.436342+0000","last_fullsized":"2024-04-25T19:44:11.436342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:17:26.264206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043125+0000","last_change":"2024-04-25T19:44:10.058621+0000","last_active":"2024-04-25T19:44:11.043125+0000","last_peered":"2024-04-25T19:44:11.043125+0000","last_clean":"2024-04-25T19:44:11.043125+0000","last_became_active":"2024-04-25T19:44:10.058237+0000","last_became_peered":"2024-04-25T19:44:10.058237+0000","last_unstale":"2024-04-25T19:44:11.043125+0000","last_undegraded":"2024-04-25T19:44:11.043125+0000","last_fullsized":"2024-04-25T19:44:11.043125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:18.399916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745612+0000","last_change":"2024-04-25T19:44:09.054405+0000","last_active":"2024-04-25T19:44:11.745612+0000","last_peered":"2024-04-25T19:44:11.745612+0000","last_clean":"2024-04-25T19:44:11.745612+0000","last_became_active":"2024-04-25T19:44:09.054065+0000","last_became_peered":"2024-04-25T19:44:09.054065+0000","last_unstale":"2024-04-25T19:44:11.745612+0000","last_undegraded":"2024-04-25T19:44:11.745612+0000","last_fullsized":"2024-04-25T19:44:11.745612+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:59.352345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043210+0000","last_change":"2024-04-25T19:44:09.047923+0000","last_active":"2024-04-25T19:44:11.043210+0000","last_peered":"2024-04-25T19:44:11.043210+0000","last_clean":"2024-04-25T19:44:11.043210+0000","last_became_active":"2024-04-25T19:44:09.047605+0000","last_became_peered":"2024-04-25T19:44:09.047605+0000","last_unstale":"2024-04-25T19:44:11.043210+0000","last_undegraded":"2024-04-25T19:44:11.043210+0000","last_fullsized":"2024-04-25T19:44:11.043210+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:30:43.807694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436073+0000","last_change":"2024-04-25T19:44:10.067620+0000","last_active":"2024-04-25T19:44:11.436073+0000","last_peered":"2024-04-25T19:44:11.436073+0000","last_clean":"2024-04-25T19:44:11.436073+0000","last_became_active":"2024-04-25T19:44:10.066364+0000","last_became_peered":"2024-04-25T19:44:10.066364+0000","last_unstale":"2024-04-25T19:44:11.436073+0000","last_undegraded":"2024-04-25T19:44:11.436073+0000","last_fullsized":"2024-04-25T19:44:11.436073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:20:30.629784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043379+0000","last_change":"2024-04-25T19:44:09.045871+0000","last_active":"2024-04-25T19:44:11.043379+0000","last_peered":"2024-04-25T19:44:11.043379+0000","last_clean":"2024-04-25T19:44:11.043379+0000","last_became_active":"2024-04-25T19:44:09.045017+0000","last_became_peered":"2024-04-25T19:44:09.045017+0000","last_unstale":"2024-04-25T19:44:11.043379+0000","last_undegraded":"2024-04-25T19:44:11.043379+0000","last_fullsized":"2024-04-25T19:44:11.043379+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:51:27.578108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.744790+0000","last_change":"2024-04-25T19:44:10.049695+0000","last_active":"2024-04-25T19:44:11.744790+0000","last_peered":"2024-04-25T19:44:11.744790+0000","last_clean":"2024-04-25T19:44:11.744790+0000","last_became_active":"2024-04-25T19:44:10.049361+0000","last_became_peered":"2024-04-25T19:44:10.049361+0000","last_unstale":"2024-04-25T19:44:11.744790+0000","last_undegraded":"2024-04-25T19:44:11.744790+0000","last_fullsized":"2024-04-25T19:44:11.744790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:57.562802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436867+0000","last_change":"2024-04-25T19:44:09.059742+0000","last_active":"2024-04-25T19:44:11.436867+0000","last_peered":"2024-04-25T19:44:11.436867+0000","last_clean":"2024-04-25T19:44:11.436867+0000","last_became_active":"2024-04-25T19:44:09.059318+0000","last_became_peered":"2024-04-25T19:44:09.059318+0000","last_unstale":"2024-04-25T19:44:11.436867+0000","last_undegraded":"2024-04-25T19:44:11.436867+0000","last_fullsized":"2024-04-25T19:44:11.436867+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:55.152664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043138+0000","last_change":"2024-04-25T19:44:10.056446+0000","last_active":"2024-04-25T19:44:11.043138+0000","last_peered":"2024-04-25T19:44:11.043138+0000","last_clean":"2024-04-25T19:44:11.043138+0000","last_became_active":"2024-04-25T19:44:10.056096+0000","last_became_peered":"2024-04-25T19:44:10.056096+0000","last_unstale":"2024-04-25T19:44:11.043138+0000","last_undegraded":"2024-04-25T19:44:11.043138+0000","last_fullsized":"2024-04-25T19:44:11.043138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:26:17.991276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436305+0000","last_change":"2024-04-25T19:44:09.054439+0000","last_active":"2024-04-25T19:44:11.436305+0000","last_peered":"2024-04-25T19:44:11.436305+0000","last_clean":"2024-04-25T19:44:11.436305+0000","last_became_active":"2024-04-25T19:44:09.053941+0000","last_became_peered":"2024-04-25T19:44:09.053941+0000","last_unstale":"2024-04-25T19:44:11.436305+0000","last_undegraded":"2024-04-25T19:44:11.436305+0000","last_fullsized":"2024-04-25T19:44:11.436305+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:55:44.329801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043164+0000","last_change":"2024-04-25T19:44:10.047455+0000","last_active":"2024-04-25T19:44:11.043164+0000","last_peered":"2024-04-25T19:44:11.043164+0000","last_clean":"2024-04-25T19:44:11.043164+0000","last_became_active":"2024-04-25T19:44:10.046766+0000","last_became_peered":"2024-04-25T19:44:10.046766+0000","last_unstale":"2024-04-25T19:44:11.043164+0000","last_undegraded":"2024-04-25T19:44:11.043164+0000","last_fullsized":"2024-04-25T19:44:11.043164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:46.058675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042351+0000","last_change":"2024-04-25T19:44:09.053423+0000","last_active":"2024-04-25T19:44:11.042351+0000","last_peered":"2024-04-25T19:44:11.042351+0000","last_clean":"2024-04-25T19:44:11.042351+0000","last_became_active":"2024-04-25T19:44:09.053138+0000","last_became_peered":"2024-04-25T19:44:09.053138+0000","last_unstale":"2024-04-25T19:44:11.042351+0000","last_undegraded":"2024-04-25T19:44:11.042351+0000","last_fullsized":"2024-04-25T19:44:11.042351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:11:22.034588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745419+0000","last_change":"2024-04-25T19:44:10.061890+0000","last_active":"2024-04-25T19:44:11.745419+0000","last_peered":"2024-04-25T19:44:11.745419+0000","last_clean":"2024-04-25T19:44:11.745419+0000","last_became_active":"2024-04-25T19:44:10.061525+0000","last_became_peered":"2024-04-25T19:44:10.061525+0000","last_unstale":"2024-04-25T19:44:11.745419+0000","last_undegraded":"2024-04-25T19:44:11.745419+0000","last_fullsized":"2024-04-25T19:44:11.745419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:33:29.948604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435794+0000","last_change":"2024-04-25T19:44:09.058395+0000","last_active":"2024-04-25T19:44:11.435794+0000","last_peered":"2024-04-25T19:44:11.435794+0000","last_clean":"2024-04-25T19:44:11.435794+0000","last_became_active":"2024-04-25T19:44:09.057411+0000","last_became_peered":"2024-04-25T19:44:09.057411+0000","last_unstale":"2024-04-25T19:44:11.435794+0000","last_undegraded":"2024-04-25T19:44:11.435794+0000","last_fullsized":"2024-04-25T19:44:11.435794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:53:44.798992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436721+0000","last_change":"2024-04-25T19:44:10.062688+0000","last_active":"2024-04-25T19:44:11.436721+0000","last_peered":"2024-04-25T19:44:11.436721+0000","last_clean":"2024-04-25T19:44:11.436721+0000","last_became_active":"2024-04-25T19:44:10.059124+0000","last_became_peered":"2024-04-25T19:44:10.059124+0000","last_unstale":"2024-04-25T19:44:11.436721+0000","last_undegraded":"2024-04-25T19:44:11.436721+0000","last_fullsized":"2024-04-25T19:44:11.436721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:18:06.928672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436491+0000","last_change":"2024-04-25T19:44:09.056965+0000","last_active":"2024-04-25T19:44:11.436491+0000","last_peered":"2024-04-25T19:44:11.436491+0000","last_clean":"2024-04-25T19:44:11.436491+0000","last_became_active":"2024-04-25T19:44:09.056557+0000","last_became_peered":"2024-04-25T19:44:09.056557+0000","last_unstale":"2024-04-25T19:44:11.436491+0000","last_undegraded":"2024-04-25T19:44:11.436491+0000","last_fullsized":"2024-04-25T19:44:11.436491+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:08:50.165109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043306+0000","last_change":"2024-04-25T19:44:10.066632+0000","last_active":"2024-04-25T19:44:11.043306+0000","last_peered":"2024-04-25T19:44:11.043306+0000","last_clean":"2024-04-25T19:44:11.043306+0000","last_became_active":"2024-04-25T19:44:10.066329+0000","last_became_peered":"2024-04-25T19:44:10.066329+0000","last_unstale":"2024-04-25T19:44:11.043306+0000","last_undegraded":"2024-04-25T19:44:11.043306+0000","last_fullsized":"2024-04-25T19:44:11.043306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:18.399916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435984+0000","last_change":"2024-04-25T19:44:09.058583+0000","last_active":"2024-04-25T19:44:11.435984+0000","last_peered":"2024-04-25T19:44:11.435984+0000","last_clean":"2024-04-25T19:44:11.435984+0000","last_became_active":"2024-04-25T19:44:09.057289+0000","last_became_peered":"2024-04-25T19:44:09.057289+0000","last_unstale":"2024-04-25T19:44:11.435984+0000","last_undegraded":"2024-04-25T19:44:11.435984+0000","last_fullsized":"2024-04-25T19:44:11.435984+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:43:22.288077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745913+0000","last_change":"2024-04-25T19:44:10.069026+0000","last_active":"2024-04-25T19:44:11.745913+0000","last_peered":"2024-04-25T19:44:11.745913+0000","last_clean":"2024-04-25T19:44:11.745913+0000","last_became_active":"2024-04-25T19:44:10.067900+0000","last_became_peered":"2024-04-25T19:44:10.067900+0000","last_unstale":"2024-04-25T19:44:11.745913+0000","last_undegraded":"2024-04-25T19:44:11.745913+0000","last_fullsized":"2024-04-25T19:44:11.745913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:51.252629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'72","reported_seq":116,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745679+0000","last_change":"2024-04-25T19:44:00.045570+0000","last_active":"2024-04-25T19:44:11.745679+0000","last_peered":"2024-04-25T19:44:11.745679+0000","last_clean":"2024-04-25T19:44:11.745679+0000","last_became_active":"2024-04-25T19:44:00.044414+0000","last_became_peered":"2024-04-25T19:44:00.044414+0000","last_unstale":"2024-04-25T19:44:11.745679+0000","last_undegraded":"2024-04-25T19:44:11.745679+0000","last_fullsized":"2024-04-25T19:44:11.745679+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:43:58.854910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:43:58.854910+0000","last_clean_scrub_stamp":"2024-04-25T19:43:58.854910+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:36.813186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436422+0000","last_change":"2024-04-25T19:44:04.907674+0000","last_active":"2024-04-25T19:44:11.436422+0000","last_peered":"2024-04-25T19:44:11.436422+0000","last_clean":"2024-04-25T19:44:11.436422+0000","last_became_active":"2024-04-25T19:44:04.907259+0000","last_became_peered":"2024-04-25T19:44:04.907259+0000","last_unstale":"2024-04-25T19:44:11.436422+0000","last_undegraded":"2024-04-25T19:44:11.436422+0000","last_fullsized":"2024-04-25T19:44:11.436422+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:58.023878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436111+0000","last_change":"2024-04-25T19:44:09.056946+0000","last_active":"2024-04-25T19:44:11.436111+0000","last_peered":"2024-04-25T19:44:11.436111+0000","last_clean":"2024-04-25T19:44:11.436111+0000","last_became_active":"2024-04-25T19:44:09.056544+0000","last_became_peered":"2024-04-25T19:44:09.056544+0000","last_unstale":"2024-04-25T19:44:11.436111+0000","last_undegraded":"2024-04-25T19:44:11.436111+0000","last_fullsized":"2024-04-25T19:44:11.436111+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:24:46.807744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436901+0000","last_change":"2024-04-25T19:44:10.060919+0000","last_active":"2024-04-25T19:44:11.436901+0000","last_peered":"2024-04-25T19:44:11.436901+0000","last_clean":"2024-04-25T19:44:11.436901+0000","last_became_active":"2024-04-25T19:44:10.059529+0000","last_became_peered":"2024-04-25T19:44:10.059529+0000","last_unstale":"2024-04-25T19:44:11.436901+0000","last_undegraded":"2024-04-25T19:44:11.436901+0000","last_fullsized":"2024-04-25T19:44:11.436901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:29.509488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436146+0000","last_change":"2024-04-25T19:44:04.904786+0000","last_active":"2024-04-25T19:44:11.436146+0000","last_peered":"2024-04-25T19:44:11.436146+0000","last_clean":"2024-04-25T19:44:11.436146+0000","last_became_active":"2024-04-25T19:44:04.904414+0000","last_became_peered":"2024-04-25T19:44:04.904414+0000","last_unstale":"2024-04-25T19:44:11.436146+0000","last_undegraded":"2024-04-25T19:44:11.436146+0000","last_fullsized":"2024-04-25T19:44:11.436146+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:48:36.719208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.041861+0000","last_change":"2024-04-25T19:44:09.040846+0000","last_active":"2024-04-25T19:44:11.041861+0000","last_peered":"2024-04-25T19:44:11.041861+0000","last_clean":"2024-04-25T19:44:11.041861+0000","last_became_active":"2024-04-25T19:44:09.040461+0000","last_became_peered":"2024-04-25T19:44:09.040461+0000","last_unstale":"2024-04-25T19:44:11.041861+0000","last_undegraded":"2024-04-25T19:44:11.041861+0000","last_fullsized":"2024-04-25T19:44:11.041861+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:48:47.178567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436718+0000","last_change":"2024-04-25T19:44:10.062165+0000","last_active":"2024-04-25T19:44:11.436718+0000","last_peered":"2024-04-25T19:44:11.436718+0000","last_clean":"2024-04-25T19:44:11.436718+0000","last_became_active":"2024-04-25T19:44:10.061626+0000","last_became_peered":"2024-04-25T19:44:10.061626+0000","last_unstale":"2024-04-25T19:44:11.436718+0000","last_undegraded":"2024-04-25T19:44:11.436718+0000","last_fullsized":"2024-04-25T19:44:11.436718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:03:01.981027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042646+0000","last_change":"2024-04-25T19:44:04.904405+0000","last_active":"2024-04-25T19:44:11.042646+0000","last_peered":"2024-04-25T19:44:11.042646+0000","last_clean":"2024-04-25T19:44:11.042646+0000","last_became_active":"2024-04-25T19:44:04.904045+0000","last_became_peered":"2024-04-25T19:44:04.904045+0000","last_unstale":"2024-04-25T19:44:11.042646+0000","last_undegraded":"2024-04-25T19:44:11.042646+0000","last_fullsized":"2024-04-25T19:44:11.042646+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:01:59.962699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.041684+0000","last_change":"2024-04-25T19:44:09.045036+0000","last_active":"2024-04-25T19:44:11.041684+0000","last_peered":"2024-04-25T19:44:11.041684+0000","last_clean":"2024-04-25T19:44:11.041684+0000","last_became_active":"2024-04-25T19:44:09.044686+0000","last_became_peered":"2024-04-25T19:44:09.044686+0000","last_unstale":"2024-04-25T19:44:11.041684+0000","last_undegraded":"2024-04-25T19:44:11.041684+0000","last_fullsized":"2024-04-25T19:44:11.041684+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:41:37.111120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042963+0000","last_change":"2024-04-25T19:44:10.053021+0000","last_active":"2024-04-25T19:44:11.042963+0000","last_peered":"2024-04-25T19:44:11.042963+0000","last_clean":"2024-04-25T19:44:11.042963+0000","last_became_active":"2024-04-25T19:44:10.052663+0000","last_became_peered":"2024-04-25T19:44:10.052663+0000","last_unstale":"2024-04-25T19:44:11.042963+0000","last_undegraded":"2024-04-25T19:44:11.042963+0000","last_fullsized":"2024-04-25T19:44:11.042963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:46.058675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436124+0000","last_change":"2024-04-25T19:44:04.907093+0000","last_active":"2024-04-25T19:44:11.436124+0000","last_peered":"2024-04-25T19:44:11.436124+0000","last_clean":"2024-04-25T19:44:11.436124+0000","last_became_active":"2024-04-25T19:44:04.906787+0000","last_became_peered":"2024-04-25T19:44:04.906787+0000","last_unstale":"2024-04-25T19:44:11.436124+0000","last_undegraded":"2024-04-25T19:44:11.436124+0000","last_fullsized":"2024-04-25T19:44:11.436124+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:28:59.374809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043241+0000","last_change":"2024-04-25T19:44:09.043484+0000","last_active":"2024-04-25T19:44:11.043241+0000","last_peered":"2024-04-25T19:44:11.043241+0000","last_clean":"2024-04-25T19:44:11.043241+0000","last_became_active":"2024-04-25T19:44:09.043155+0000","last_became_peered":"2024-04-25T19:44:09.043155+0000","last_unstale":"2024-04-25T19:44:11.043241+0000","last_undegraded":"2024-04-25T19:44:11.043241+0000","last_fullsized":"2024-04-25T19:44:11.043241+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:25.996441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043040+0000","last_change":"2024-04-25T19:44:10.050042+0000","last_active":"2024-04-25T19:44:11.043040+0000","last_peered":"2024-04-25T19:44:11.043040+0000","last_clean":"2024-04-25T19:44:11.043040+0000","last_became_active":"2024-04-25T19:44:10.049701+0000","last_became_peered":"2024-04-25T19:44:10.049701+0000","last_unstale":"2024-04-25T19:44:11.043040+0000","last_undegraded":"2024-04-25T19:44:11.043040+0000","last_fullsized":"2024-04-25T19:44:11.043040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:52:52.432869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043083+0000","last_change":"2024-04-25T19:44:09.040543+0000","last_active":"2024-04-25T19:44:11.043083+0000","last_peered":"2024-04-25T19:44:11.043083+0000","last_clean":"2024-04-25T19:44:11.043083+0000","last_became_active":"2024-04-25T19:44:09.040137+0000","last_became_peered":"2024-04-25T19:44:09.040137+0000","last_unstale":"2024-04-25T19:44:11.043083+0000","last_undegraded":"2024-04-25T19:44:11.043083+0000","last_fullsized":"2024-04-25T19:44:11.043083+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:59:23.033455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043278+0000","last_change":"2024-04-25T19:44:04.905598+0000","last_active":"2024-04-25T19:44:11.043278+0000","last_peered":"2024-04-25T19:44:11.043278+0000","last_clean":"2024-04-25T19:44:11.043278+0000","last_became_active":"2024-04-25T19:44:04.904778+0000","last_became_peered":"2024-04-25T19:44:04.904778+0000","last_unstale":"2024-04-25T19:44:11.043278+0000","last_undegraded":"2024-04-25T19:44:11.043278+0000","last_fullsized":"2024-04-25T19:44:11.043278+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:03:40.381610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745207+0000","last_change":"2024-04-25T19:44:10.063439+0000","last_active":"2024-04-25T19:44:11.745207+0000","last_peered":"2024-04-25T19:44:11.745207+0000","last_clean":"2024-04-25T19:44:11.745207+0000","last_became_active":"2024-04-25T19:44:10.062610+0000","last_became_peered":"2024-04-25T19:44:10.062610+0000","last_unstale":"2024-04-25T19:44:11.745207+0000","last_undegraded":"2024-04-25T19:44:11.745207+0000","last_fullsized":"2024-04-25T19:44:11.745207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:43:53.239881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043105+0000","last_change":"2024-04-25T19:44:09.044556+0000","last_active":"2024-04-25T19:44:11.043105+0000","last_peered":"2024-04-25T19:44:11.043105+0000","last_clean":"2024-04-25T19:44:11.043105+0000","last_became_active":"2024-04-25T19:44:09.044205+0000","last_became_peered":"2024-04-25T19:44:09.044205+0000","last_unstale":"2024-04-25T19:44:11.043105+0000","last_undegraded":"2024-04-25T19:44:11.043105+0000","last_fullsized":"2024-04-25T19:44:11.043105+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:59:23.033455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436963+0000","last_change":"2024-04-25T19:44:04.904842+0000","last_active":"2024-04-25T19:44:11.436963+0000","last_peered":"2024-04-25T19:44:11.436963+0000","last_clean":"2024-04-25T19:44:11.436963+0000","last_became_active":"2024-04-25T19:44:04.904449+0000","last_became_peered":"2024-04-25T19:44:04.904449+0000","last_unstale":"2024-04-25T19:44:11.436963+0000","last_undegraded":"2024-04-25T19:44:11.436963+0000","last_fullsized":"2024-04-25T19:44:11.436963+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:38.322685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.744965+0000","last_change":"2024-04-25T19:44:10.060164+0000","last_active":"2024-04-25T19:44:11.744965+0000","last_peered":"2024-04-25T19:44:11.744965+0000","last_clean":"2024-04-25T19:44:11.744965+0000","last_became_active":"2024-04-25T19:44:10.059272+0000","last_became_peered":"2024-04-25T19:44:10.059272+0000","last_unstale":"2024-04-25T19:44:11.744965+0000","last_undegraded":"2024-04-25T19:44:11.744965+0000","last_fullsized":"2024-04-25T19:44:11.744965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:49:57.885441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436380+0000","last_change":"2024-04-25T19:44:09.058552+0000","last_active":"2024-04-25T19:44:11.436380+0000","last_peered":"2024-04-25T19:44:11.436380+0000","last_clean":"2024-04-25T19:44:11.436380+0000","last_became_active":"2024-04-25T19:44:09.058107+0000","last_became_peered":"2024-04-25T19:44:09.058107+0000","last_unstale":"2024-04-25T19:44:11.436380+0000","last_undegraded":"2024-04-25T19:44:11.436380+0000","last_fullsized":"2024-04-25T19:44:11.436380+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:27.990801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042931+0000","last_change":"2024-04-25T19:44:04.904847+0000","last_active":"2024-04-25T19:44:11.042931+0000","last_peered":"2024-04-25T19:44:11.042931+0000","last_clean":"2024-04-25T19:44:11.042931+0000","last_became_active":"2024-04-25T19:44:04.904399+0000","last_became_peered":"2024-04-25T19:44:04.904399+0000","last_unstale":"2024-04-25T19:44:11.042931+0000","last_undegraded":"2024-04-25T19:44:11.042931+0000","last_fullsized":"2024-04-25T19:44:11.042931+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:31.155970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435636+0000","last_change":"2024-04-25T19:44:10.067351+0000","last_active":"2024-04-25T19:44:11.435636+0000","last_peered":"2024-04-25T19:44:11.435636+0000","last_clean":"2024-04-25T19:44:11.435636+0000","last_became_active":"2024-04-25T19:44:10.065833+0000","last_became_peered":"2024-04-25T19:44:10.065833+0000","last_unstale":"2024-04-25T19:44:11.435636+0000","last_undegraded":"2024-04-25T19:44:11.435636+0000","last_fullsized":"2024-04-25T19:44:11.435636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:41:44.189493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745703+0000","last_change":"2024-04-25T19:44:09.048708+0000","last_active":"2024-04-25T19:44:11.745703+0000","last_peered":"2024-04-25T19:44:11.745703+0000","last_clean":"2024-04-25T19:44:11.745703+0000","last_became_active":"2024-04-25T19:44:09.048320+0000","last_became_peered":"2024-04-25T19:44:09.048320+0000","last_unstale":"2024-04-25T19:44:11.745703+0000","last_undegraded":"2024-04-25T19:44:11.745703+0000","last_fullsized":"2024-04-25T19:44:11.745703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:48:47.178567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436571+0000","last_change":"2024-04-25T19:44:04.907462+0000","last_active":"2024-04-25T19:44:11.436571+0000","last_peered":"2024-04-25T19:44:11.436571+0000","last_clean":"2024-04-25T19:44:11.436571+0000","last_became_active":"2024-04-25T19:44:04.907134+0000","last_became_peered":"2024-04-25T19:44:04.907134+0000","last_unstale":"2024-04-25T19:44:11.436571+0000","last_undegraded":"2024-04-25T19:44:11.436571+0000","last_fullsized":"2024-04-25T19:44:11.436571+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:03.881071+0000","last_clean_scrub_stamp":"2024-04-25T19:44:03.881071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:05:53.288205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042736+0000","last_change":"2024-04-25T19:44:10.062982+0000","last_active":"2024-04-25T19:44:11.042736+0000","last_peered":"2024-04-25T19:44:11.042736+0000","last_clean":"2024-04-25T19:44:11.042736+0000","last_became_active":"2024-04-25T19:44:10.062674+0000","last_became_peered":"2024-04-25T19:44:10.062674+0000","last_unstale":"2024-04-25T19:44:11.042736+0000","last_undegraded":"2024-04-25T19:44:11.042736+0000","last_fullsized":"2024-04-25T19:44:11.042736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:21.909648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043372+0000","last_change":"2024-04-25T19:44:10.056196+0000","last_active":"2024-04-25T19:44:11.043372+0000","last_peered":"2024-04-25T19:44:11.043372+0000","last_clean":"2024-04-25T19:44:11.043372+0000","last_became_active":"2024-04-25T19:44:10.055892+0000","last_became_peered":"2024-04-25T19:44:10.055892+0000","last_unstale":"2024-04-25T19:44:11.043372+0000","last_undegraded":"2024-04-25T19:44:11.043372+0000","last_fullsized":"2024-04-25T19:44:11.043372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:40.201502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436060+0000","last_change":"2024-04-25T19:44:09.056570+0000","last_active":"2024-04-25T19:44:11.436060+0000","last_peered":"2024-04-25T19:44:11.436060+0000","last_clean":"2024-04-25T19:44:11.436060+0000","last_became_active":"2024-04-25T19:44:09.055464+0000","last_became_peered":"2024-04-25T19:44:09.055464+0000","last_unstale":"2024-04-25T19:44:11.436060+0000","last_undegraded":"2024-04-25T19:44:11.436060+0000","last_fullsized":"2024-04-25T19:44:11.436060+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:48:00.496337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745172+0000","last_change":"2024-04-25T19:44:10.063199+0000","last_active":"2024-04-25T19:44:11.745172+0000","last_peered":"2024-04-25T19:44:11.745172+0000","last_clean":"2024-04-25T19:44:11.745172+0000","last_became_active":"2024-04-25T19:44:10.062127+0000","last_became_peered":"2024-04-25T19:44:10.062127+0000","last_unstale":"2024-04-25T19:44:11.745172+0000","last_undegraded":"2024-04-25T19:44:11.745172+0000","last_fullsized":"2024-04-25T19:44:11.745172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:36:33.677454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042922+0000","last_change":"2024-04-25T19:44:09.059522+0000","last_active":"2024-04-25T19:44:11.042922+0000","last_peered":"2024-04-25T19:44:11.042922+0000","last_clean":"2024-04-25T19:44:11.042922+0000","last_became_active":"2024-04-25T19:44:09.059182+0000","last_became_peered":"2024-04-25T19:44:09.059182+0000","last_unstale":"2024-04-25T19:44:11.042922+0000","last_undegraded":"2024-04-25T19:44:11.042922+0000","last_fullsized":"2024-04-25T19:44:11.042922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:25.699599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043346+0000","last_change":"2024-04-25T19:44:09.054779+0000","last_active":"2024-04-25T19:44:11.043346+0000","last_peered":"2024-04-25T19:44:11.043346+0000","last_clean":"2024-04-25T19:44:11.043346+0000","last_became_active":"2024-04-25T19:44:09.054451+0000","last_became_peered":"2024-04-25T19:44:09.054451+0000","last_unstale":"2024-04-25T19:44:11.043346+0000","last_undegraded":"2024-04-25T19:44:11.043346+0000","last_fullsized":"2024-04-25T19:44:11.043346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:59.352345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.041751+0000","last_change":"2024-04-25T19:44:10.043921+0000","last_active":"2024-04-25T19:44:11.041751+0000","last_peered":"2024-04-25T19:44:11.041751+0000","last_clean":"2024-04-25T19:44:11.041751+0000","last_became_active":"2024-04-25T19:44:10.043528+0000","last_became_peered":"2024-04-25T19:44:10.043528+0000","last_unstale":"2024-04-25T19:44:11.041751+0000","last_undegraded":"2024-04-25T19:44:11.041751+0000","last_fullsized":"2024-04-25T19:44:11.041751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:55:45.331783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745974+0000","last_change":"2024-04-25T19:44:09.055027+0000","last_active":"2024-04-25T19:44:11.745974+0000","last_peered":"2024-04-25T19:44:11.745974+0000","last_clean":"2024-04-25T19:44:11.745974+0000","last_became_active":"2024-04-25T19:44:09.054691+0000","last_became_peered":"2024-04-25T19:44:09.054691+0000","last_unstale":"2024-04-25T19:44:11.745974+0000","last_undegraded":"2024-04-25T19:44:11.745974+0000","last_fullsized":"2024-04-25T19:44:11.745974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:25.996441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435340+0000","last_change":"2024-04-25T19:44:10.067089+0000","last_active":"2024-04-25T19:44:11.435340+0000","last_peered":"2024-04-25T19:44:11.435340+0000","last_clean":"2024-04-25T19:44:11.435340+0000","last_became_active":"2024-04-25T19:44:10.065329+0000","last_became_peered":"2024-04-25T19:44:10.065329+0000","last_unstale":"2024-04-25T19:44:11.435340+0000","last_undegraded":"2024-04-25T19:44:11.435340+0000","last_fullsized":"2024-04-25T19:44:11.435340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:51.252629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436700+0000","last_change":"2024-04-25T19:44:10.060616+0000","last_active":"2024-04-25T19:44:11.436700+0000","last_peered":"2024-04-25T19:44:11.436700+0000","last_clean":"2024-04-25T19:44:11.436700+0000","last_became_active":"2024-04-25T19:44:10.060285+0000","last_became_peered":"2024-04-25T19:44:10.060285+0000","last_unstale":"2024-04-25T19:44:11.436700+0000","last_undegraded":"2024-04-25T19:44:11.436700+0000","last_fullsized":"2024-04-25T19:44:11.436700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:23:38.854656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435812+0000","last_change":"2024-04-25T19:44:09.059263+0000","last_active":"2024-04-25T19:44:11.435812+0000","last_peered":"2024-04-25T19:44:11.435812+0000","last_clean":"2024-04-25T19:44:11.435812+0000","last_became_active":"2024-04-25T19:44:09.058936+0000","last_became_peered":"2024-04-25T19:44:09.058936+0000","last_unstale":"2024-04-25T19:44:11.435812+0000","last_undegraded":"2024-04-25T19:44:11.435812+0000","last_fullsized":"2024-04-25T19:44:11.435812+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:49.562956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042882+0000","last_change":"2024-04-25T19:44:10.051152+0000","last_active":"2024-04-25T19:44:11.042882+0000","last_peered":"2024-04-25T19:44:11.042882+0000","last_clean":"2024-04-25T19:44:11.042882+0000","last_became_active":"2024-04-25T19:44:10.050607+0000","last_became_peered":"2024-04-25T19:44:10.050607+0000","last_unstale":"2024-04-25T19:44:11.042882+0000","last_undegraded":"2024-04-25T19:44:11.042882+0000","last_fullsized":"2024-04-25T19:44:11.042882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:28.992782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.744879+0000","last_change":"2024-04-25T19:44:09.054711+0000","last_active":"2024-04-25T19:44:11.744879+0000","last_peered":"2024-04-25T19:44:11.744879+0000","last_clean":"2024-04-25T19:44:11.744879+0000","last_became_active":"2024-04-25T19:44:09.053989+0000","last_became_peered":"2024-04-25T19:44:09.053989+0000","last_unstale":"2024-04-25T19:44:11.744879+0000","last_undegraded":"2024-04-25T19:44:11.744879+0000","last_fullsized":"2024-04-25T19:44:11.744879+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:25.699599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436322+0000","last_change":"2024-04-25T19:44:10.059956+0000","last_active":"2024-04-25T19:44:11.436322+0000","last_peered":"2024-04-25T19:44:11.436322+0000","last_clean":"2024-04-25T19:44:11.436322+0000","last_became_active":"2024-04-25T19:44:10.058700+0000","last_became_peered":"2024-04-25T19:44:10.058700+0000","last_unstale":"2024-04-25T19:44:11.436322+0000","last_undegraded":"2024-04-25T19:44:11.436322+0000","last_fullsized":"2024-04-25T19:44:11.436322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:25:09.001237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745309+0000","last_change":"2024-04-25T19:44:09.044537+0000","last_active":"2024-04-25T19:44:11.745309+0000","last_peered":"2024-04-25T19:44:11.745309+0000","last_clean":"2024-04-25T19:44:11.745309+0000","last_became_active":"2024-04-25T19:44:09.044205+0000","last_became_peered":"2024-04-25T19:44:09.044205+0000","last_unstale":"2024-04-25T19:44:11.745309+0000","last_undegraded":"2024-04-25T19:44:11.745309+0000","last_fullsized":"2024-04-25T19:44:11.745309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:11:22.034588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042844+0000","last_change":"2024-04-25T19:44:10.050365+0000","last_active":"2024-04-25T19:44:11.042844+0000","last_peered":"2024-04-25T19:44:11.042844+0000","last_clean":"2024-04-25T19:44:11.042844+0000","last_became_active":"2024-04-25T19:44:10.049730+0000","last_became_peered":"2024-04-25T19:44:10.049730+0000","last_unstale":"2024-04-25T19:44:11.042844+0000","last_undegraded":"2024-04-25T19:44:11.042844+0000","last_fullsized":"2024-04-25T19:44:11.042844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:26:17.991276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436186+0000","last_change":"2024-04-25T19:44:09.058547+0000","last_active":"2024-04-25T19:44:11.436186+0000","last_peered":"2024-04-25T19:44:11.436186+0000","last_clean":"2024-04-25T19:44:11.436186+0000","last_became_active":"2024-04-25T19:44:09.057348+0000","last_became_peered":"2024-04-25T19:44:09.057348+0000","last_unstale":"2024-04-25T19:44:11.436186+0000","last_undegraded":"2024-04-25T19:44:11.436186+0000","last_fullsized":"2024-04-25T19:44:11.436186+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:38:52.992542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042693+0000","last_change":"2024-04-25T19:44:10.047756+0000","last_active":"2024-04-25T19:44:11.042693+0000","last_peered":"2024-04-25T19:44:11.042693+0000","last_clean":"2024-04-25T19:44:11.042693+0000","last_became_active":"2024-04-25T19:44:10.047155+0000","last_became_peered":"2024-04-25T19:44:10.047155+0000","last_unstale":"2024-04-25T19:44:11.042693+0000","last_undegraded":"2024-04-25T19:44:11.042693+0000","last_fullsized":"2024-04-25T19:44:11.042693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:43:23.290058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745329+0000","last_change":"2024-04-25T19:44:09.053455+0000","last_active":"2024-04-25T19:44:11.745329+0000","last_peered":"2024-04-25T19:44:11.745329+0000","last_clean":"2024-04-25T19:44:11.745329+0000","last_became_active":"2024-04-25T19:44:09.053118+0000","last_became_peered":"2024-04-25T19:44:09.053118+0000","last_unstale":"2024-04-25T19:44:11.745329+0000","last_undegraded":"2024-04-25T19:44:11.745329+0000","last_fullsized":"2024-04-25T19:44:11.745329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:59:23.033455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.043396+0000","last_change":"2024-04-25T19:44:10.047937+0000","last_active":"2024-04-25T19:44:11.043396+0000","last_peered":"2024-04-25T19:44:11.043396+0000","last_clean":"2024-04-25T19:44:11.043396+0000","last_became_active":"2024-04-25T19:44:10.046231+0000","last_became_peered":"2024-04-25T19:44:10.046231+0000","last_unstale":"2024-04-25T19:44:11.043396+0000","last_undegraded":"2024-04-25T19:44:11.043396+0000","last_fullsized":"2024-04-25T19:44:11.043396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:50.564937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.436779+0000","last_change":"2024-04-25T19:44:09.060536+0000","last_active":"2024-04-25T19:44:11.436779+0000","last_peered":"2024-04-25T19:44:11.436779+0000","last_clean":"2024-04-25T19:44:11.436779+0000","last_became_active":"2024-04-25T19:44:09.059473+0000","last_became_peered":"2024-04-25T19:44:09.059473+0000","last_unstale":"2024-04-25T19:44:11.436779+0000","last_undegraded":"2024-04-25T19:44:11.436779+0000","last_fullsized":"2024-04-25T19:44:11.436779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:20.907667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.435968+0000","last_change":"2024-04-25T19:44:10.043795+0000","last_active":"2024-04-25T19:44:11.435968+0000","last_peered":"2024-04-25T19:44:11.435968+0000","last_clean":"2024-04-25T19:44:11.435968+0000","last_became_active":"2024-04-25T19:44:10.043342+0000","last_became_peered":"2024-04-25T19:44:10.043342+0000","last_unstale":"2024-04-25T19:44:11.435968+0000","last_undegraded":"2024-04-25T19:44:11.435968+0000","last_fullsized":"2024-04-25T19:44:11.435968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:52:49.026097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042861+0000","last_change":"2024-04-25T19:44:09.044583+0000","last_active":"2024-04-25T19:44:11.042861+0000","last_peered":"2024-04-25T19:44:11.042861+0000","last_clean":"2024-04-25T19:44:11.042861+0000","last_became_active":"2024-04-25T19:44:09.044225+0000","last_became_peered":"2024-04-25T19:44:09.044225+0000","last_unstale":"2024-04-25T19:44:11.042861+0000","last_undegraded":"2024-04-25T19:44:11.042861+0000","last_fullsized":"2024-04-25T19:44:11.042861+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:25.699599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.042456+0000","last_change":"2024-04-25T19:44:10.049048+0000","last_active":"2024-04-25T19:44:11.042456+0000","last_peered":"2024-04-25T19:44:11.042456+0000","last_clean":"2024-04-25T19:44:11.042456+0000","last_became_active":"2024-04-25T19:44:10.048740+0000","last_became_peered":"2024-04-25T19:44:10.048740+0000","last_unstale":"2024-04-25T19:44:11.042456+0000","last_undegraded":"2024-04-25T19:44:11.042456+0000","last_fullsized":"2024-04-25T19:44:11.042456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:09.017642+0000","last_clean_scrub_stamp":"2024-04-25T19:44:09.017642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:19.654106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T19:44:11.745972+0000","last_change":"2024-04-25T19:44:09.046229+0000","last_active":"2024-04-25T19:44:11.745972+0000","last_peered":"2024-04-25T19:44:11.745972+0000","last_clean":"2024-04-25T19:44:11.745972+0000","last_became_active":"2024-04-25T19:44:09.045423+0000","last_became_peered":"2024-04-25T19:44:09.045423+0000","last_unstale":"2024-04-25T19:44:11.745972+0000","last_undegraded":"2024-04-25T19:44:11.745972+0000","last_fullsized":"2024-04-25T19:44:11.745972+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:44:08.015661+0000","last_clean_scrub_stamp":"2024-04-25T19:44:08.015661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:30:43.807694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672967,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":64550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672967,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":654870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672967,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":790528,"data_stored":653786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":188416,"data_stored":63326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T19:44:18.437 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T19:44:18.437 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T19:44:18.437 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-25T19:44:18.437 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T19:44:18.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.582+0000 7f0d894e6640 1 Processor -- start 2024-04-25T19:44:18.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.582+0000 7f0d894e6640 1 -- start start 2024-04-25T19:44:18.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.583+0000 7f0d894e6640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0d840a56d0 0x7f0d840acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:18.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.583+0000 7f0d894e6640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0d840ad280 0x7f0d841513f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:18.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.583+0000 7f0d894e6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d84146c60 con 0x7f0d8406da90 2024-04-25T19:44:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.583+0000 7f0d894e6640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f0d840ad9d0 con 0x7f0d840ad280 2024-04-25T19:44:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.583+0000 7f0d894e6640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f0d840ad690 con 0x7f0d840a56d0 2024-04-25T19:44:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.583+0000 7f0d827fc640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0d840a56d0 0x7f0d840acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.583+0000 7f0d82ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0d8406da90 0x7f0d841472d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52918/0 (socket says 172.21.15.29:52918) 2024-04-25T19:44:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.583+0000 7f0d82ffd640 1 -- 172.21.15.29:0/2942450145 learned_addr learned my addr 172.21.15.29:0/2942450145 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.583+0000 7f0d837fe640 1 --2- 172.21.15.29:0/2942450145 >> v2:172.21.15.102:3300/0 conn(0x7f0d840ad280 0x7f0d841513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.583+0000 7f0d81ffb640 1 -- 172.21.15.29:0/2942450145 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2248501725 0 0) 0x7f0d84146c60 con 0x7f0d8406da90 2024-04-25T19:44:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.583+0000 7f0d81ffb640 1 -- 172.21.15.29:0/2942450145 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d6c003520 con 0x7f0d8406da90 2024-04-25T19:44:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.584+0000 7f0d827fc640 1 -- 172.21.15.29:0/2942450145 >> v2:172.21.15.102:3300/0 conn(0x7f0d840ad280 msgr2=0x7f0d841513f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.584+0000 7f0d827fc640 1 --2- 172.21.15.29:0/2942450145 >> v2:172.21.15.102:3300/0 conn(0x7f0d840ad280 0x7f0d841513f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.584+0000 7f0d827fc640 1 -- 172.21.15.29:0/2942450145 >> v1:172.21.15.29:6789/0 conn(0x7f0d8406da90 legacy=0x7f0d841472d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.584+0000 7f0d827fc640 1 -- 172.21.15.29:0/2942450145 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d841479e0 con 0x7f0d840a56d0 2024-04-25T19:44:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.584+0000 7f0d837fe640 1 --2- 172.21.15.29:0/2942450145 >> v2:172.21.15.102:3300/0 conn(0x7f0d840ad280 0x7f0d841513f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:44:18.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.584+0000 7f0d827fc640 1 --2- 172.21.15.29:0/2942450145 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0d840a56d0 0x7f0d840acd40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0d780097a0 tx=0x7f0d7802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4b4ca6bffdb31e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:18.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.584+0000 7f0d81ffb640 1 -- 172.21.15.29:0/2942450145 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d78034070 con 0x7f0d840a56d0 2024-04-25T19:44:18.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.584+0000 7f0d81ffb640 1 -- 172.21.15.29:0/2942450145 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d78030610 con 0x7f0d840a56d0 2024-04-25T19:44:18.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.584+0000 7f0d81ffb640 1 -- 172.21.15.29:0/2942450145 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d780309e0 con 0x7f0d840a56d0 2024-04-25T19:44:18.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.585+0000 7f0d894e6640 1 -- 172.21.15.29:0/2942450145 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0d840a56d0 msgr2=0x7f0d840acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:18.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.585+0000 7f0d894e6640 1 --2- 172.21.15.29:0/2942450145 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0d840a56d0 0x7f0d840acd40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0d780097a0 tx=0x7f0d7802b960 comp rx=0 tx=0).stop 2024-04-25T19:44:18.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.585+0000 7f0d894e6640 1 -- 172.21.15.29:0/2942450145 shutdown_connections 2024-04-25T19:44:18.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.585+0000 7f0d894e6640 1 --2- 172.21.15.29:0/2942450145 >> v2:172.21.15.102:3300/0 conn(0x7f0d840ad280 0x7f0d841513f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.585+0000 7f0d894e6640 1 --2- 172.21.15.29:0/2942450145 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0d840a56d0 0x7f0d840acd40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.585+0000 7f0d894e6640 1 -- 172.21.15.29:0/2942450145 >> 172.21.15.29:0/2942450145 conn(0x7f0d840bc3b0 msgr2=0x7f0d840bc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:18.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.585+0000 7f0d894e6640 1 -- 172.21.15.29:0/2942450145 shutdown_connections 2024-04-25T19:44:18.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.585+0000 7f0d894e6640 1 -- 172.21.15.29:0/2942450145 wait complete. 2024-04-25T19:44:18.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.587+0000 7f0d894e6640 1 Processor -- start 2024-04-25T19:44:18.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.587+0000 7f0d894e6640 1 -- start start 2024-04-25T19:44:18.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.588+0000 7f0d894e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d8406da90 0x7f0d84142270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.588+0000 7f0d894e6640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0d840a56d0 0x7f0d8414a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.588+0000 7f0d894e6640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d840ad280 0x7f0d841e0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.588+0000 7f0d894e6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0d84151ab0 con 0x7f0d8406da90 2024-04-25T19:44:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.588+0000 7f0d894e6640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0d84151930 con 0x7f0d840ad280 2024-04-25T19:44:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.588+0000 7f0d894e6640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f0d84151c30 con 0x7f0d840a56d0 2024-04-25T19:44:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.588+0000 7f0d82ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d8406da90 0x7f0d84142270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.588+0000 7f0d827fc640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0d840a56d0 0x7f0d8414a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.588+0000 7f0d827fc640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0d840a56d0 0x7f0d8414a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:35104/0 (socket says 172.21.15.29:35104) 2024-04-25T19:44:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.588+0000 7f0d82ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d8406da90 0x7f0d84142270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39748/0 (socket says 172.21.15.29:39748) 2024-04-25T19:44:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.588+0000 7f0d827fc640 1 -- 172.21.15.29:0/643158511 learned_addr learned my addr 172.21.15.29:0/643158511 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.588+0000 7f0d837fe640 1 --2- 172.21.15.29:0/643158511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d840ad280 0x7f0d841e0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.588+0000 7f0d827fc640 1 -- 172.21.15.29:0/643158511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d840ad280 msgr2=0x7f0d841e0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.588+0000 7f0d827fc640 1 --2- 172.21.15.29:0/643158511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d840ad280 0x7f0d841e0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.588+0000 7f0d827fc640 1 -- 172.21.15.29:0/643158511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d8406da90 msgr2=0x7f0d84142270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.588+0000 7f0d827fc640 1 --2- 172.21.15.29:0/643158511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d8406da90 0x7f0d84142270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.588+0000 7f0d827fc640 1 -- 172.21.15.29:0/643158511 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d8414c190 con 0x7f0d840a56d0 2024-04-25T19:44:18.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.588+0000 7f0d837fe640 1 --2- 172.21.15.29:0/643158511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d840ad280 0x7f0d841e0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:44:18.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.588+0000 7f0d82ffd640 1 --2- 172.21.15.29:0/643158511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d8406da90 0x7f0d84142270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:18.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.588+0000 7f0d827fc640 1 --2- 172.21.15.29:0/643158511 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0d840a56d0 0x7f0d8414a020 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0d7802bf70 tx=0x7f0d780046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:18.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.589+0000 7f0d6b7fe640 1 -- 172.21.15.29:0/643158511 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d78034040 con 0x7f0d840a56d0 2024-04-25T19:44:18.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.589+0000 7f0d6b7fe640 1 -- 172.21.15.29:0/643158511 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d7803aaa0 con 0x7f0d840a56d0 2024-04-25T19:44:18.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.589+0000 7f0d894e6640 1 -- 172.21.15.29:0/643158511 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d84149490 con 0x7f0d840a56d0 2024-04-25T19:44:18.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.589+0000 7f0d6b7fe640 1 -- 172.21.15.29:0/643158511 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d780443f0 con 0x7f0d840a56d0 2024-04-25T19:44:18.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.589+0000 7f0d894e6640 1 -- 172.21.15.29:0/643158511 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d8413c140 con 0x7f0d840a56d0 2024-04-25T19:44:18.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.589+0000 7f0d894e6640 1 -- 172.21.15.29:0/643158511 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d840507b0 con 0x7f0d840a56d0 2024-04-25T19:44:18.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.592+0000 7f0d6b7fe640 1 -- 172.21.15.29:0/643158511 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f0d7804b060 con 0x7f0d840a56d0 2024-04-25T19:44:18.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.592+0000 7f0d6b7fe640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:18.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.592+0000 7f0d6b7fe640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:18.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.592+0000 7f0d6b7fe640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:18.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.592+0000 7f0d6b7fe640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:18.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.592+0000 7f0d6b7fe640 1 --2- 172.21.15.29:0/643158511 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f0d700ac560 0x7f0d700aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:18.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.593+0000 7f0d6b7fe640 1 -- 172.21.15.29:0/643158511 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f0d780fb410 con 0x7f0d840a56d0 2024-04-25T19:44:18.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.593+0000 7f0d82ffd640 1 --2- 172.21.15.29:0/643158511 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f0d700ac560 0x7f0d700aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:18.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.593+0000 7f0d82ffd640 1 --2- 172.21.15.29:0/643158511 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f0d700ac560 0x7f0d700aea70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0d74005880 tx=0x7f0d74009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:18.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.596+0000 7f0d6b7fe640 1 -- 172.21.15.29:0/643158511 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0d7807fb80 con 0x7f0d840a56d0 2024-04-25T19:44:18.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.817+0000 7f0d894e6640 1 -- 172.21.15.29:0/643158511 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f0d840ba1d0 con 0x7f0d840a56d0 2024-04-25T19:44:18.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.818+0000 7f0d6b7fe640 1 -- 172.21.15.29:0/643158511 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f0d7803f070 con 0x7f0d840a56d0 2024-04-25T19:44:18.819 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:18.819 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T19:44:18.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.820+0000 7f0d894e6640 10 mgrc shutdown 2024-04-25T19:44:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.820+0000 7f0d894e6640 1 -- 172.21.15.29:0/643158511 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f0d700ac560 msgr2=0x7f0d700aea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.820+0000 7f0d894e6640 1 --2- 172.21.15.29:0/643158511 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f0d700ac560 0x7f0d700aea70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0d74005880 tx=0x7f0d74009040 comp rx=0 tx=0).stop 2024-04-25T19:44:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.820+0000 7f0d894e6640 1 -- 172.21.15.29:0/643158511 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0d840a56d0 msgr2=0x7f0d8414a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.820+0000 7f0d894e6640 1 --2- 172.21.15.29:0/643158511 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0d840a56d0 0x7f0d8414a020 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0d7802bf70 tx=0x7f0d780046c0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.820+0000 7f0d894e6640 1 -- 172.21.15.29:0/643158511 shutdown_connections 2024-04-25T19:44:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.820+0000 7f0d894e6640 1 --2- 172.21.15.29:0/643158511 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f0d700ac560 0x7f0d700aea70 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.820+0000 7f0d894e6640 1 --2- 172.21.15.29:0/643158511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d840ad280 0x7f0d841e0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.820+0000 7f0d894e6640 1 --2- 172.21.15.29:0/643158511 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0d840a56d0 0x7f0d8414a020 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.820+0000 7f0d894e6640 1 --2- 172.21.15.29:0/643158511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d8406da90 0x7f0d84142270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.820+0000 7f0d894e6640 1 -- 172.21.15.29:0/643158511 >> 172.21.15.29:0/643158511 conn(0x7f0d840bc3b0 msgr2=0x7f0d840bb610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.820+0000 7f0d894e6640 1 -- 172.21.15.29:0/643158511 shutdown_connections 2024-04-25T19:44:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.820+0000 7f0d894e6640 1 -- 172.21.15.29:0/643158511 wait complete. 2024-04-25T19:44:18.832 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-25T19:44:18.832 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:44:18.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.942+0000 7f6efddb6640 1 Processor -- start 2024-04-25T19:44:18.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.942+0000 7f6efddb6640 1 -- start start 2024-04-25T19:44:18.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.943+0000 7f6efddb6640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ef8143e30 0x7f6ef8144240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:18.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.943+0000 7f6efddb6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6ef80aceb0 0x7f6ef8151590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:18.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.943+0000 7f6efddb6640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ef80a5a90 con 0x7f6ef80acaa0 2024-04-25T19:44:18.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.943+0000 7f6efddb6640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f6ef8147790 con 0x7f6ef80aceb0 2024-04-25T19:44:18.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.943+0000 7f6efddb6640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f6ef8147a50 con 0x7f6ef8143e30 2024-04-25T19:44:18.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.944+0000 7f6ef77fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ef8143e30 0x7f6ef8144240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:18.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.944+0000 7f6ef7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6ef80aceb0 0x7f6ef8151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:18.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.944+0000 7f6ef77fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ef8143e30 0x7f6ef8144240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:35108/0 (socket says 172.21.15.29:35108) 2024-04-25T19:44:18.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.944+0000 7f6ef7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6ef80aceb0 0x7f6ef8151590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39762/0 (socket says 172.21.15.29:39762) 2024-04-25T19:44:18.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.944+0000 7f6ef77fe640 1 -- 172.21.15.29:0/2680543569 learned_addr learned my addr 172.21.15.29:0/2680543569 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:18.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.944+0000 7f6ef77fe640 1 -- 172.21.15.29:0/2680543569 >> v2:172.21.15.29:3300/0 conn(0x7f6ef80aceb0 msgr2=0x7f6ef8151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:18.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.944+0000 7f6ef77fe640 1 --2- 172.21.15.29:0/2680543569 >> v2:172.21.15.29:3300/0 conn(0x7f6ef80aceb0 0x7f6ef8151590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.944+0000 7f6ef77fe640 1 -- 172.21.15.29:0/2680543569 >> v1:172.21.15.102:6789/0 conn(0x7f6ef80acaa0 legacy=0x7f6ef8146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:18.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.944+0000 7f6ef77fe640 1 -- 172.21.15.29:0/2680543569 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef80ad2c0 con 0x7f6ef8143e30 2024-04-25T19:44:18.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.944+0000 7f6ef7fff640 1 --2- 172.21.15.29:0/2680543569 >> v2:172.21.15.29:3300/0 conn(0x7f6ef80aceb0 0x7f6ef8151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:18.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.945+0000 7f6ef77fe640 1 --2- 172.21.15.29:0/2680543569 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ef8143e30 0x7f6ef8144240 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6ee00098a0 tx=0x7f6ee002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=958d96cbaf1a5feb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:18.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.945+0000 7f6ef67fc640 1 -- 172.21.15.29:0/2680543569 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee0034070 con 0x7f6ef8143e30 2024-04-25T19:44:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.945+0000 7f6ef67fc640 1 -- 172.21.15.29:0/2680543569 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ee0030840 con 0x7f6ef8143e30 2024-04-25T19:44:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.945+0000 7f6ef67fc640 1 -- 172.21.15.29:0/2680543569 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee0030c10 con 0x7f6ef8143e30 2024-04-25T19:44:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.945+0000 7f6efddb6640 1 -- 172.21.15.29:0/2680543569 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ef8143e30 msgr2=0x7f6ef8144240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.945+0000 7f6efddb6640 1 --2- 172.21.15.29:0/2680543569 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ef8143e30 0x7f6ef8144240 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6ee00098a0 tx=0x7f6ee002b980 comp rx=0 tx=0).stop 2024-04-25T19:44:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.946+0000 7f6efddb6640 1 -- 172.21.15.29:0/2680543569 shutdown_connections 2024-04-25T19:44:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.946+0000 7f6efddb6640 1 --2- 172.21.15.29:0/2680543569 >> v2:172.21.15.29:3300/0 conn(0x7f6ef80aceb0 0x7f6ef8151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.946+0000 7f6efddb6640 1 --2- 172.21.15.29:0/2680543569 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ef8143e30 0x7f6ef8144240 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.946+0000 7f6efddb6640 1 -- 172.21.15.29:0/2680543569 >> 172.21.15.29:0/2680543569 conn(0x7f6ef80aafc0 msgr2=0x7f6ef80ab3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.946+0000 7f6efddb6640 1 -- 172.21.15.29:0/2680543569 shutdown_connections 2024-04-25T19:44:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.946+0000 7f6efddb6640 1 -- 172.21.15.29:0/2680543569 wait complete. 2024-04-25T19:44:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.947+0000 7f6efddb6640 1 Processor -- start 2024-04-25T19:44:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.947+0000 7f6efddb6640 1 -- start start 2024-04-25T19:44:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.948+0000 7f6efddb6640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ef80acaa0 0x7f6ef814bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.948+0000 7f6efddb6640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ef80aceb0 0x7f6ef814ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.948+0000 7f6efddb6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ef8143e30 0x7f6ef81494d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.948+0000 7f6efddb6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef8151ad0 con 0x7f6ef8143e30 2024-04-25T19:44:18.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.948+0000 7f6efddb6640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef8147530 con 0x7f6ef80acaa0 2024-04-25T19:44:18.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.948+0000 7f6efddb6640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f6ef8151c50 con 0x7f6ef80aceb0 2024-04-25T19:44:18.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.948+0000 7f6ef6ffd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ef80aceb0 0x7f6ef814ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:18.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.948+0000 7f6ef7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ef8143e30 0x7f6ef81494d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:18.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.948+0000 7f6ef7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ef8143e30 0x7f6ef81494d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39766/0 (socket says 172.21.15.29:39766) 2024-04-25T19:44:18.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.948+0000 7f6ef7fff640 1 -- 172.21.15.29:0/144368252 learned_addr learned my addr 172.21.15.29:0/144368252 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:18.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.948+0000 7f6ef77fe640 1 --2- 172.21.15.29:0/144368252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ef80acaa0 0x7f6ef814bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:18.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.949+0000 7f6ef7fff640 1 -- 172.21.15.29:0/144368252 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ef80aceb0 msgr2=0x7f6ef814ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:18.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.949+0000 7f6ef7fff640 1 --2- 172.21.15.29:0/144368252 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ef80aceb0 0x7f6ef814ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.949+0000 7f6ef7fff640 1 -- 172.21.15.29:0/144368252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ef80acaa0 msgr2=0x7f6ef814bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.949+0000 7f6ef7fff640 1 --2- 172.21.15.29:0/144368252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ef80acaa0 0x7f6ef814bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.949+0000 7f6ef7fff640 1 -- 172.21.15.29:0/144368252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef814a1a0 con 0x7f6ef8143e30 2024-04-25T19:44:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.949+0000 7f6ef6ffd640 1 --2- 172.21.15.29:0/144368252 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ef80aceb0 0x7f6ef814ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.949+0000 7f6ef7fff640 1 --2- 172.21.15.29:0/144368252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ef8143e30 0x7f6ef81494d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6ef000efb0 tx=0x7f6ef000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.949+0000 7f6eeffff640 1 -- 172.21.15.29:0/144368252 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0018020 con 0x7f6ef8143e30 2024-04-25T19:44:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.949+0000 7f6eeffff640 1 -- 172.21.15.29:0/144368252 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ef0014480 con 0x7f6ef8143e30 2024-04-25T19:44:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.949+0000 7f6eeffff640 1 -- 172.21.15.29:0/144368252 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0014850 con 0x7f6ef8143e30 2024-04-25T19:44:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.949+0000 7f6efddb6640 1 -- 172.21.15.29:0/144368252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ef8146560 con 0x7f6ef8143e30 2024-04-25T19:44:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.949+0000 7f6efddb6640 1 -- 172.21.15.29:0/144368252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ef814a8a0 con 0x7f6ef8143e30 2024-04-25T19:44:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.950+0000 7f6efddb6640 1 -- 172.21.15.29:0/144368252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ef81dd4e0 con 0x7f6ef8143e30 2024-04-25T19:44:18.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.951+0000 7f6eeffff640 1 -- 172.21.15.29:0/144368252 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f6ef0005d10 con 0x7f6ef8143e30 2024-04-25T19:44:18.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.951+0000 7f6eeffff640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:18.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.952+0000 7f6eeffff640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:18.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.952+0000 7f6eeffff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:18.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.952+0000 7f6eeffff640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:18.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.952+0000 7f6eeffff640 1 --2- 172.21.15.29:0/144368252 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f6ed80ac560 0x7f6ed80aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:18.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.952+0000 7f6eeffff640 1 -- 172.21.15.29:0/144368252 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f6ef00da590 con 0x7f6ef8143e30 2024-04-25T19:44:18.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.952+0000 7f6ef77fe640 1 --2- 172.21.15.29:0/144368252 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f6ed80ac560 0x7f6ed80aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:18.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.953+0000 7f6ef77fe640 1 --2- 172.21.15.29:0/144368252 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f6ed80ac560 0x7f6ed80aea70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0036000 tx=0x7f6ee0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:18.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:18.956+0000 7f6eeffff640 1 -- 172.21.15.29:0/144368252 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6ef0022030 con 0x7f6ef8143e30 2024-04-25T19:44:19.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.144+0000 7f6efddb6640 1 -- 172.21.15.29:0/144368252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6ef814c2a0 con 0x7f6ef8143e30 2024-04-25T19:44:19.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.145+0000 7f6eeffff640 1 -- 172.21.15.29:0/144368252 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+13512 (secure 0 0 0) 0x7f6ef0010040 con 0x7f6ef8143e30 2024-04-25T19:44:19.146 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:19.146 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"1b3c39b5-e787-409e-b0bd-f4f491c89d72","created":"2024-04-25T19:43:47.854697+0000","modified":"2024-04-25T19:44:11.035019+0000","last_up_change":"2024-04-25T19:43:57.853237+0000","last_in_change":"2024-04-25T19:43:49.022776+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:43:58.096700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T19:44:03.617011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:44:07.890493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2024-04-25T19:44:08.355570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d805223-ee84-494f-bd31-59091906aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6801","nonce":695704147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6803","nonce":695704147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6807","nonce":695704147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6805","nonce":695704147}]},"public_addr":"172.21.15.29:6801/695704147","cluster_addr":"172.21.15.29:6803/695704147","heartbeat_back_addr":"172.21.15.29:6807/695704147","heartbeat_front_addr":"172.21.15.29:6805/695704147","state":["exists","up"]},{"osd":1,"uuid":"607c5038-38af-440f-8371-e71758549920","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6814","nonce":3135821452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6817","nonce":3135821452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6823","nonce":3135821452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6821","nonce":3135821452}]},"public_addr":"172.21.15.102:6814/3135821452","cluster_addr":"172.21.15.102:6817/3135821452","heartbeat_back_addr":"172.21.15.102:6823/3135821452","heartbeat_front_addr":"172.21.15.102:6821/3135821452","state":["exists","up"]},{"osd":2,"uuid":"6859ae76-54bb-4c55-b0dc-5e2735c0d31c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6801","nonce":2071844144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6803","nonce":2071844144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6811","nonce":2071844144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6806","nonce":2071844144}]},"public_addr":"172.21.15.102:6801/2071844144","cluster_addr":"172.21.15.102:6803/2071844144","heartbeat_back_addr":"172.21.15.102:6811/2071844144","heartbeat_front_addr":"172.21.15.102:6806/2071844144","state":["exists","up"]},{"osd":3,"uuid":"a98b7b88-5206-4f22-84a6-78212e0d5f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6805","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6807","nonce":339944984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6810","nonce":339944984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6820","nonce":339944984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6815","nonce":339944984}]},"public_addr":"172.21.15.102:6807/339944984","cluster_addr":"172.21.15.102:6810/339944984","heartbeat_back_addr":"172.21.15.102:6820/339944984","heartbeat_front_addr":"172.21.15.102:6815/339944984","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.121529+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.201155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.264387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.198713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:44:19.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.147+0000 7f6efddb6640 10 mgrc shutdown 2024-04-25T19:44:19.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.147+0000 7f6efddb6640 1 -- 172.21.15.29:0/144368252 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f6ed80ac560 msgr2=0x7f6ed80aea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:19.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.147+0000 7f6efddb6640 1 --2- 172.21.15.29:0/144368252 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f6ed80ac560 0x7f6ed80aea70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0036000 tx=0x7f6ee0008030 comp rx=0 tx=0).stop 2024-04-25T19:44:19.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.147+0000 7f6efddb6640 1 -- 172.21.15.29:0/144368252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ef8143e30 msgr2=0x7f6ef81494d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:19.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.147+0000 7f6efddb6640 1 --2- 172.21.15.29:0/144368252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ef8143e30 0x7f6ef81494d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6ef000efb0 tx=0x7f6ef000c490 comp rx=0 tx=0).stop 2024-04-25T19:44:19.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.147+0000 7f6efddb6640 1 -- 172.21.15.29:0/144368252 shutdown_connections 2024-04-25T19:44:19.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.147+0000 7f6efddb6640 1 --2- 172.21.15.29:0/144368252 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f6ed80ac560 0x7f6ed80aea70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.147+0000 7f6efddb6640 1 --2- 172.21.15.29:0/144368252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ef8143e30 0x7f6ef81494d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.147+0000 7f6efddb6640 1 --2- 172.21.15.29:0/144368252 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ef80aceb0 0x7f6ef814ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.147+0000 7f6efddb6640 1 --2- 172.21.15.29:0/144368252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ef80acaa0 0x7f6ef814bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.147+0000 7f6efddb6640 1 -- 172.21.15.29:0/144368252 >> 172.21.15.29:0/144368252 conn(0x7f6ef80aafc0 msgr2=0x7f6ef81431f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:19.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.147+0000 7f6efddb6640 1 -- 172.21.15.29:0/144368252 shutdown_connections 2024-04-25T19:44:19.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.147+0000 7f6efddb6640 1 -- 172.21.15.29:0/144368252 wait complete. 2024-04-25T19:44:19.159 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:43:58.096700+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '13', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4, 'score_stable': 4, 'optimal_score': 0.5, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-25T19:44:03.617011+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T19:44:07.890493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T19:44:08.355570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T19:44:19.160 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:44:19.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.272+0000 7fac6e26a640 1 Processor -- start 2024-04-25T19:44:19.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.272+0000 7fac6e26a640 1 -- start start 2024-04-25T19:44:19.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.273+0000 7fac6e26a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fac68143e30 0x7fac68144240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:19.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.273+0000 7fac6e26a640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fac680aceb0 0x7fac68151590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:19.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.273+0000 7fac6e26a640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac680a5a90 con 0x7fac680acaa0 2024-04-25T19:44:19.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.273+0000 7fac6e26a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fac68147790 con 0x7fac68143e30 2024-04-25T19:44:19.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.273+0000 7fac6e26a640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fac68147a50 con 0x7fac680aceb0 2024-04-25T19:44:19.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.273+0000 7fac67fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fac68143e30 0x7fac68144240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:19.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.273+0000 7fac67fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fac68143e30 0x7fac68144240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39778/0 (socket says 172.21.15.29:39778) 2024-04-25T19:44:19.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.273+0000 7fac67fff640 1 -- 172.21.15.29:0/2383730729 learned_addr learned my addr 172.21.15.29:0/2383730729 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:19.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.273+0000 7fac6c807640 1 --2- 172.21.15.29:0/2383730729 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fac680aceb0 0x7fac68151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:19.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.273+0000 7fac67fff640 1 -- 172.21.15.29:0/2383730729 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fac680aceb0 msgr2=0x7fac68151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:19.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.273+0000 7fac67fff640 1 --2- 172.21.15.29:0/2383730729 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fac680aceb0 0x7fac68151590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.273+0000 7fac67fff640 1 -- 172.21.15.29:0/2383730729 >> v1:172.21.15.102:6789/0 conn(0x7fac680acaa0 legacy=0x7fac68146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:19.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.273+0000 7fac67fff640 1 -- 172.21.15.29:0/2383730729 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac680ad2c0 con 0x7fac68143e30 2024-04-25T19:44:19.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.273+0000 7fac6c807640 1 --2- 172.21.15.29:0/2383730729 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fac680aceb0 0x7fac68151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:19.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.274+0000 7fac67fff640 1 --2- 172.21.15.29:0/2383730729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac68143e30 0x7fac68144240 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fac58009870 tx=0x7fac5802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed465401935a76c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:19.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.274+0000 7fac66ffd640 1 -- 172.21.15.29:0/2383730729 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac58034070 con 0x7fac68143e30 2024-04-25T19:44:19.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.274+0000 7fac66ffd640 1 -- 172.21.15.29:0/2383730729 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac58030840 con 0x7fac68143e30 2024-04-25T19:44:19.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.274+0000 7fac66ffd640 1 -- 172.21.15.29:0/2383730729 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac58030c10 con 0x7fac68143e30 2024-04-25T19:44:19.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.275+0000 7fac6e26a640 1 -- 172.21.15.29:0/2383730729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac68143e30 msgr2=0x7fac68144240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:19.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.275+0000 7fac6e26a640 1 --2- 172.21.15.29:0/2383730729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac68143e30 0x7fac68144240 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fac58009870 tx=0x7fac5802b980 comp rx=0 tx=0).stop 2024-04-25T19:44:19.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.275+0000 7fac6e26a640 1 -- 172.21.15.29:0/2383730729 shutdown_connections 2024-04-25T19:44:19.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.275+0000 7fac6e26a640 1 --2- 172.21.15.29:0/2383730729 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fac680aceb0 0x7fac68151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.275+0000 7fac6e26a640 1 --2- 172.21.15.29:0/2383730729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac68143e30 0x7fac68144240 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.275+0000 7fac6e26a640 1 -- 172.21.15.29:0/2383730729 >> 172.21.15.29:0/2383730729 conn(0x7fac680aafc0 msgr2=0x7fac680ab3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:19.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.275+0000 7fac6e26a640 1 -- 172.21.15.29:0/2383730729 shutdown_connections 2024-04-25T19:44:19.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.275+0000 7fac6e26a640 1 -- 172.21.15.29:0/2383730729 wait complete. 2024-04-25T19:44:19.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.276+0000 7fac6e26a640 1 Processor -- start 2024-04-25T19:44:19.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.277+0000 7fac6e26a640 1 -- start start 2024-04-25T19:44:19.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac6e26a640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac680acaa0 0x7fac6814aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:19.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac6e26a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac680aceb0 0x7fac6814d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:19.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac6e26a640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fac68143e30 0x7fac6814c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:19.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac6e26a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fac68151ad0 con 0x7fac680aceb0 2024-04-25T19:44:19.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac6e26a640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fac68147530 con 0x7fac680acaa0 2024-04-25T19:44:19.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac6e26a640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fac68151c50 con 0x7fac68143e30 2024-04-25T19:44:19.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac677fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac680aceb0 0x7fac6814d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:19.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac6c807640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fac68143e30 0x7fac6814c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac6c807640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fac68143e30 0x7fac6814c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:35134/0 (socket says 172.21.15.29:35134) 2024-04-25T19:44:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac677fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac680aceb0 0x7fac6814d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39780/0 (socket says 172.21.15.29:39780) 2024-04-25T19:44:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac6c807640 1 -- 172.21.15.29:0/4200892938 learned_addr learned my addr 172.21.15.29:0/4200892938 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac67fff640 1 --2- 172.21.15.29:0/4200892938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac680acaa0 0x7fac6814aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac677fe640 1 -- 172.21.15.29:0/4200892938 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fac68143e30 msgr2=0x7fac6814c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac677fe640 1 --2- 172.21.15.29:0/4200892938 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fac68143e30 0x7fac6814c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac677fe640 1 -- 172.21.15.29:0/4200892938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac680acaa0 msgr2=0x7fac6814aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac677fe640 1 --2- 172.21.15.29:0/4200892938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac680acaa0 0x7fac6814aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac677fe640 1 -- 172.21.15.29:0/4200892938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac6814afd0 con 0x7fac680aceb0 2024-04-25T19:44:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac6c807640 1 --2- 172.21.15.29:0/4200892938 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fac68143e30 0x7fac6814c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac677fe640 1 --2- 172.21.15.29:0/4200892938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac680aceb0 0x7fac6814d5e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fac5c00c390 tx=0x7fac5c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac64ff9640 1 -- 172.21.15.29:0/4200892938 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac5c017070 con 0x7fac680aceb0 2024-04-25T19:44:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac64ff9640 1 -- 172.21.15.29:0/4200892938 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac5c0052f0 con 0x7fac680aceb0 2024-04-25T19:44:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.278+0000 7fac6e26a640 1 -- 172.21.15.29:0/4200892938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac68148000 con 0x7fac680aceb0 2024-04-25T19:44:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.279+0000 7fac64ff9640 1 -- 172.21.15.29:0/4200892938 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac5c003850 con 0x7fac680aceb0 2024-04-25T19:44:19.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.279+0000 7fac6e26a640 1 -- 172.21.15.29:0/4200892938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac6814c8e0 con 0x7fac680aceb0 2024-04-25T19:44:19.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.279+0000 7fac6e26a640 1 -- 172.21.15.29:0/4200892938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac680bb710 con 0x7fac680aceb0 2024-04-25T19:44:19.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.281+0000 7fac64ff9640 1 -- 172.21.15.29:0/4200892938 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7fac5c01a070 con 0x7fac680aceb0 2024-04-25T19:44:19.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.281+0000 7fac64ff9640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:19.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.281+0000 7fac64ff9640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:19.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.281+0000 7fac64ff9640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:19.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.281+0000 7fac64ff9640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:19.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.282+0000 7fac64ff9640 1 --2- 172.21.15.29:0/4200892938 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fac540ac530 0x7fac540aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:19.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.282+0000 7fac64ff9640 1 -- 172.21.15.29:0/4200892938 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7fac5c0d86b0 con 0x7fac680aceb0 2024-04-25T19:44:19.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.282+0000 7fac67fff640 1 --2- 172.21.15.29:0/4200892938 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fac540ac530 0x7fac540aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:19.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.282+0000 7fac67fff640 1 --2- 172.21.15.29:0/4200892938 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fac540ac530 0x7fac540aea40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fac58036000 tx=0x7fac58008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:19.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.285+0000 7fac64ff9640 1 -- 172.21.15.29:0/4200892938 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fac5c021080 con 0x7fac680aceb0 2024-04-25T19:44:19.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.468+0000 7fac6e26a640 1 -- 172.21.15.29:0/4200892938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fac680507b0 con 0x7fac680aceb0 2024-04-25T19:44:19.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.468+0000 7fac64ff9640 1 -- 172.21.15.29:0/4200892938 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7fac5c0a2cd0 con 0x7fac680aceb0 2024-04-25T19:44:19.470 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-04-25T19:44:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.470+0000 7fac6e26a640 10 mgrc shutdown 2024-04-25T19:44:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.470+0000 7fac6e26a640 1 -- 172.21.15.29:0/4200892938 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fac540ac530 msgr2=0x7fac540aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.470+0000 7fac6e26a640 1 --2- 172.21.15.29:0/4200892938 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fac540ac530 0x7fac540aea40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fac58036000 tx=0x7fac58008030 comp rx=0 tx=0).stop 2024-04-25T19:44:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.470+0000 7fac6e26a640 1 -- 172.21.15.29:0/4200892938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac680aceb0 msgr2=0x7fac6814d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.470+0000 7fac6e26a640 1 --2- 172.21.15.29:0/4200892938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac680aceb0 0x7fac6814d5e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fac5c00c390 tx=0x7fac5c00c850 comp rx=0 tx=0).stop 2024-04-25T19:44:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.470+0000 7fac6e26a640 1 -- 172.21.15.29:0/4200892938 shutdown_connections 2024-04-25T19:44:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.470+0000 7fac6e26a640 1 --2- 172.21.15.29:0/4200892938 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fac540ac530 0x7fac540aea40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.470+0000 7fac6e26a640 1 --2- 172.21.15.29:0/4200892938 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fac68143e30 0x7fac6814c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.470+0000 7fac6e26a640 1 --2- 172.21.15.29:0/4200892938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac680aceb0 0x7fac6814d5e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.470+0000 7fac6e26a640 1 --2- 172.21.15.29:0/4200892938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac680acaa0 0x7fac6814aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.470+0000 7fac6e26a640 1 -- 172.21.15.29:0/4200892938 >> 172.21.15.29:0/4200892938 conn(0x7fac680aafc0 msgr2=0x7fac680a96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.470+0000 7fac6e26a640 1 -- 172.21.15.29:0/4200892938 shutdown_connections 2024-04-25T19:44:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.470+0000 7fac6e26a640 1 -- 172.21.15.29:0/4200892938 wait complete. 2024-04-25T19:44:19.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-04-25T19:44:19.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.596+0000 7fae18755640 1 Processor -- start 2024-04-25T19:44:19.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.596+0000 7fae18755640 1 -- start start 2024-04-25T19:44:19.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.597+0000 7fae18755640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae101465a0 0x7fae1014a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:19.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.597+0000 7fae18755640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae101469b0 0x7fae1014aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:19.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.597+0000 7fae18755640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae100a6e30 con 0x7fae1006df20 2024-04-25T19:44:19.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.597+0000 7fae18755640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fae1006e330 con 0x7fae101465a0 2024-04-25T19:44:19.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.597+0000 7fae18755640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fae1014b1a0 con 0x7fae101469b0 2024-04-25T19:44:19.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.597+0000 7fae16cf2640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae101469b0 0x7fae1014aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:19.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.597+0000 7fae15cf0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae101465a0 0x7fae1014a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:19.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.597+0000 7fae16cf2640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae101469b0 0x7fae1014aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:35146/0 (socket says 172.21.15.29:35146) 2024-04-25T19:44:19.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.597+0000 7fae16cf2640 1 -- 172.21.15.29:0/2397404149 learned_addr learned my addr 172.21.15.29:0/2397404149 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:19.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.597+0000 7fae15cf0640 1 --2- 172.21.15.29:0/2397404149 >> v2:172.21.15.29:3300/0 conn(0x7fae101465a0 0x7fae1014a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39784/0 (socket says 172.21.15.29:39784) 2024-04-25T19:44:19.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.598+0000 7fae16cf2640 1 -- 172.21.15.29:0/2397404149 >> v2:172.21.15.29:3300/0 conn(0x7fae101465a0 msgr2=0x7fae1014a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:19.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.598+0000 7fae16cf2640 1 --2- 172.21.15.29:0/2397404149 >> v2:172.21.15.29:3300/0 conn(0x7fae101465a0 0x7fae1014a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.598+0000 7fae16cf2640 1 -- 172.21.15.29:0/2397404149 >> v1:172.21.15.102:6789/0 conn(0x7fae1006df20 legacy=0x7fae10073920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:19.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.598+0000 7fae16cf2640 1 -- 172.21.15.29:0/2397404149 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae101526a0 con 0x7fae101469b0 2024-04-25T19:44:19.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.598+0000 7fae154ef640 1 -- 172.21.15.29:0/2397404149 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 522920023 0 0) 0x7fae08001110 con 0x7fae1006df20 2024-04-25T19:44:19.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.598+0000 7fae15cf0640 1 --2- 172.21.15.29:0/2397404149 >> v2:172.21.15.29:3300/0 conn(0x7fae101465a0 0x7fae1014a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:19.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.598+0000 7fae16cf2640 1 --2- 172.21.15.29:0/2397404149 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae101469b0 0x7fae1014aa50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fae00009e50 tx=0x7fae0002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d724fb941d7ba73 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:19.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.599+0000 7fae154ef640 1 -- 172.21.15.29:0/2397404149 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0002ca70 con 0x7fae101469b0 2024-04-25T19:44:19.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.599+0000 7fae154ef640 1 -- 172.21.15.29:0/2397404149 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae0002cc10 con 0x7fae101469b0 2024-04-25T19:44:19.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.599+0000 7fae154ef640 1 -- 172.21.15.29:0/2397404149 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae000343f0 con 0x7fae101469b0 2024-04-25T19:44:19.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.599+0000 7fae18755640 1 -- 172.21.15.29:0/2397404149 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae101469b0 msgr2=0x7fae1014aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:19.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.599+0000 7fae18755640 1 --2- 172.21.15.29:0/2397404149 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae101469b0 0x7fae1014aa50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fae00009e50 tx=0x7fae0002bed0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.599+0000 7fae18755640 1 -- 172.21.15.29:0/2397404149 shutdown_connections 2024-04-25T19:44:19.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.599+0000 7fae18755640 1 --2- 172.21.15.29:0/2397404149 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae101469b0 0x7fae1014aa50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.599+0000 7fae18755640 1 --2- 172.21.15.29:0/2397404149 >> v2:172.21.15.29:3300/0 conn(0x7fae101465a0 0x7fae1014a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.599+0000 7fae18755640 1 -- 172.21.15.29:0/2397404149 >> 172.21.15.29:0/2397404149 conn(0x7fae100a9390 msgr2=0x7fae100a97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:19.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.599+0000 7fae18755640 1 -- 172.21.15.29:0/2397404149 shutdown_connections 2024-04-25T19:44:19.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.599+0000 7fae18755640 1 -- 172.21.15.29:0/2397404149 wait complete. 2024-04-25T19:44:19.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.601+0000 7fae18755640 1 Processor -- start 2024-04-25T19:44:19.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.601+0000 7fae18755640 1 -- start start 2024-04-25T19:44:19.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.602+0000 7fae18755640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae1006df20 0x7fae101496f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:19.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.602+0000 7fae18755640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae101465a0 0x7fae1013aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:19.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.602+0000 7fae18755640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae101469b0 0x7fae100b5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:19.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.602+0000 7fae18755640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fae10147290 con 0x7fae101465a0 2024-04-25T19:44:19.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.602+0000 7fae18755640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fae10147110 con 0x7fae101469b0 2024-04-25T19:44:19.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.602+0000 7fae18755640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fae10147410 con 0x7fae1006df20 2024-04-25T19:44:19.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.602+0000 7fae15cf0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae101465a0 0x7fae1013aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:19.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.602+0000 7fae164f1640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae1006df20 0x7fae101496f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:19.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.602+0000 7fae15cf0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae101465a0 0x7fae1013aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39792/0 (socket says 172.21.15.29:39792) 2024-04-25T19:44:19.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.602+0000 7fae15cf0640 1 -- 172.21.15.29:0/3990840532 learned_addr learned my addr 172.21.15.29:0/3990840532 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:19.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.602+0000 7fae16cf2640 1 --2- 172.21.15.29:0/3990840532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae101469b0 0x7fae100b5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:19.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.602+0000 7fae164f1640 1 -- 172.21.15.29:0/3990840532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae101469b0 msgr2=0x7fae100b5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:19.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.602+0000 7fae164f1640 1 --2- 172.21.15.29:0/3990840532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae101469b0 0x7fae100b5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.602+0000 7fae164f1640 1 -- 172.21.15.29:0/3990840532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae101465a0 msgr2=0x7fae1013aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:19.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.602+0000 7fae164f1640 1 --2- 172.21.15.29:0/3990840532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae101465a0 0x7fae1013aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.602+0000 7fae164f1640 1 -- 172.21.15.29:0/3990840532 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae10138a30 con 0x7fae1006df20 2024-04-25T19:44:19.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.602+0000 7fae15cf0640 1 --2- 172.21.15.29:0/3990840532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae101465a0 0x7fae1013aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:19.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.602+0000 7fae164f1640 1 --2- 172.21.15.29:0/3990840532 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae1006df20 0x7fae101496f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fae0800c410 tx=0x7fae0800c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:19.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.602+0000 7fae0effd640 1 -- 172.21.15.29:0/3990840532 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae08017070 con 0x7fae1006df20 2024-04-25T19:44:19.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.602+0000 7fae0effd640 1 -- 172.21.15.29:0/3990840532 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae080035f0 con 0x7fae1006df20 2024-04-25T19:44:19.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.603+0000 7fae0effd640 1 -- 172.21.15.29:0/3990840532 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae08003a20 con 0x7fae1006df20 2024-04-25T19:44:19.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.603+0000 7fae18755640 1 -- 172.21.15.29:0/3990840532 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae100ac080 con 0x7fae1006df20 2024-04-25T19:44:19.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.603+0000 7fae18755640 1 -- 172.21.15.29:0/3990840532 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fae100b6e90 con 0x7fae1006df20 2024-04-25T19:44:19.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.603+0000 7fae18755640 1 -- 172.21.15.29:0/3990840532 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae1013b370 con 0x7fae1006df20 2024-04-25T19:44:19.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.606+0000 7fae0effd640 1 -- 172.21.15.29:0/3990840532 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7fae0801a070 con 0x7fae1006df20 2024-04-25T19:44:19.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.606+0000 7fae0effd640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:19.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.606+0000 7fae0effd640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.606+0000 7fae0effd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.606+0000 7fae0effd640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.606+0000 7fae0effd640 1 --2- 172.21.15.29:0/3990840532 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fadfc0ac560 0x7fadfc0aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.606+0000 7fae0effd640 1 -- 172.21.15.29:0/3990840532 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7fae080d8ad0 con 0x7fae1006df20 2024-04-25T19:44:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.607+0000 7fae15cf0640 1 --2- 172.21.15.29:0/3990840532 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fadfc0ac560 0x7fadfc0aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.607+0000 7fae15cf0640 1 --2- 172.21.15.29:0/3990840532 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fadfc0ac560 0x7fadfc0aea70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fae04009580 tx=0x7fae04009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:19.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.610+0000 7fae0effd640 1 -- 172.21.15.29:0/3990840532 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fae08021020 con 0x7fae1006df20 2024-04-25T19:44:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.794+0000 7fae18755640 1 -- 172.21.15.29:0/3990840532 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7fae100507b0 con 0x7fae1006df20 2024-04-25T19:44:19.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.795+0000 7fae0effd640 1 -- 172.21.15.29:0/3990840532 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) ==== 92+0+10 (secure 0 0 0) 0x7fae100a6e30 con 0x7fae1006df20 2024-04-25T19:44:19.796 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 8 2024-04-25T19:44:19.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.797+0000 7fae18755640 10 mgrc shutdown 2024-04-25T19:44:19.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.797+0000 7fae18755640 1 -- 172.21.15.29:0/3990840532 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fadfc0ac560 msgr2=0x7fadfc0aea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:19.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.797+0000 7fae18755640 1 --2- 172.21.15.29:0/3990840532 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fadfc0ac560 0x7fadfc0aea70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fae04009580 tx=0x7fae04009190 comp rx=0 tx=0).stop 2024-04-25T19:44:19.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.797+0000 7fae18755640 1 -- 172.21.15.29:0/3990840532 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae1006df20 msgr2=0x7fae101496f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:19.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.797+0000 7fae18755640 1 --2- 172.21.15.29:0/3990840532 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae1006df20 0x7fae101496f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fae0800c410 tx=0x7fae0800c810 comp rx=0 tx=0).stop 2024-04-25T19:44:19.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.797+0000 7fae18755640 1 -- 172.21.15.29:0/3990840532 shutdown_connections 2024-04-25T19:44:19.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.797+0000 7fae18755640 1 --2- 172.21.15.29:0/3990840532 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fadfc0ac560 0x7fadfc0aea70 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.797+0000 7fae18755640 1 --2- 172.21.15.29:0/3990840532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae101469b0 0x7fae100b5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.797+0000 7fae18755640 1 --2- 172.21.15.29:0/3990840532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae101465a0 0x7fae1013aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.797+0000 7fae18755640 1 --2- 172.21.15.29:0/3990840532 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae1006df20 0x7fae101496f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.797+0000 7fae18755640 1 -- 172.21.15.29:0/3990840532 >> 172.21.15.29:0/3990840532 conn(0x7fae100a9390 msgr2=0x7fae100684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:19.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.797+0000 7fae18755640 1 -- 172.21.15.29:0/3990840532 shutdown_connections 2024-04-25T19:44:19.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.797+0000 7fae18755640 1 -- 172.21.15.29:0/3990840532 wait complete. 2024-04-25T19:44:19.810 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:44:19.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.922+0000 7fb79cd86640 1 Processor -- start 2024-04-25T19:44:19.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.922+0000 7fb79cd86640 1 -- start start 2024-04-25T19:44:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.923+0000 7fb79cd86640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb79806df40 0x7fb7980aca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.923+0000 7fb79cd86640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb7980acf70 0x7fb798151400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.923+0000 7fb79cd86640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7980ad500 con 0x7fb798146640 2024-04-25T19:44:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.924+0000 7fb79cd86640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fb7980ad7c0 con 0x7fb7980acf70 2024-04-25T19:44:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.924+0000 7fb79cd86640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fb7980ae650 con 0x7fb79806df40 2024-04-25T19:44:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.924+0000 7fb79659c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb798146640 0x7fb7980aa840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52920/0 (socket says 172.21.15.29:52920) 2024-04-25T19:44:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.924+0000 7fb79659c640 1 -- 172.21.15.29:0/4121228521 learned_addr learned my addr 172.21.15.29:0/4121228521 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.924+0000 7fb795d9b640 1 --2- 172.21.15.29:0/4121228521 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb79806df40 0x7fb7980aca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.924+0000 7fb796d9d640 1 --2- 172.21.15.29:0/4121228521 >> v2:172.21.15.102:3300/0 conn(0x7fb7980acf70 0x7fb798151400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.924+0000 7fb79559a640 1 -- 172.21.15.29:0/4121228521 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3223793166 0 0) 0x7fb7980ad500 con 0x7fb798146640 2024-04-25T19:44:19.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.924+0000 7fb79559a640 1 -- 172.21.15.29:0/4121228521 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb780003520 con 0x7fb798146640 2024-04-25T19:44:19.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.924+0000 7fb795d9b640 1 -- 172.21.15.29:0/4121228521 >> v2:172.21.15.102:3300/0 conn(0x7fb7980acf70 msgr2=0x7fb798151400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:19.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.924+0000 7fb795d9b640 1 --2- 172.21.15.29:0/4121228521 >> v2:172.21.15.102:3300/0 conn(0x7fb7980acf70 0x7fb798151400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.924+0000 7fb795d9b640 1 -- 172.21.15.29:0/4121228521 >> v1:172.21.15.29:6789/0 conn(0x7fb798146640 legacy=0x7fb7980aa840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:19.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.924+0000 7fb795d9b640 1 -- 172.21.15.29:0/4121228521 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb79806e350 con 0x7fb79806df40 2024-04-25T19:44:19.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.925+0000 7fb795d9b640 1 --2- 172.21.15.29:0/4121228521 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb79806df40 0x7fb7980aca30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb78c0097a0 tx=0x7fb78c02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2518db96718e681 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:19.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.925+0000 7fb79559a640 1 -- 172.21.15.29:0/4121228521 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb78c034070 con 0x7fb79806df40 2024-04-25T19:44:19.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.925+0000 7fb79559a640 1 -- 172.21.15.29:0/4121228521 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb78c030610 con 0x7fb79806df40 2024-04-25T19:44:19.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.925+0000 7fb79559a640 1 -- 172.21.15.29:0/4121228521 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb78c0309e0 con 0x7fb79806df40 2024-04-25T19:44:19.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.926+0000 7fb79cd86640 1 -- 172.21.15.29:0/4121228521 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb79806df40 msgr2=0x7fb7980aca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:19.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.926+0000 7fb79cd86640 1 --2- 172.21.15.29:0/4121228521 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb79806df40 0x7fb7980aca30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb78c0097a0 tx=0x7fb78c02b960 comp rx=0 tx=0).stop 2024-04-25T19:44:19.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.926+0000 7fb79cd86640 1 -- 172.21.15.29:0/4121228521 shutdown_connections 2024-04-25T19:44:19.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.926+0000 7fb79cd86640 1 --2- 172.21.15.29:0/4121228521 >> v2:172.21.15.102:3300/0 conn(0x7fb7980acf70 0x7fb798151400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.926+0000 7fb79cd86640 1 --2- 172.21.15.29:0/4121228521 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb79806df40 0x7fb7980aca30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.926+0000 7fb79cd86640 1 -- 172.21.15.29:0/4121228521 >> 172.21.15.29:0/4121228521 conn(0x7fb7980ab710 msgr2=0x7fb7980bbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:19.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.926+0000 7fb79cd86640 1 -- 172.21.15.29:0/4121228521 shutdown_connections 2024-04-25T19:44:19.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.926+0000 7fb79cd86640 1 -- 172.21.15.29:0/4121228521 wait complete. 2024-04-25T19:44:19.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.927+0000 7fb79cd86640 1 Processor -- start 2024-04-25T19:44:19.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.927+0000 7fb79cd86640 1 -- start start 2024-04-25T19:44:19.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.928+0000 7fb79cd86640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb79806df40 0x7fb798147eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:19.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.928+0000 7fb79cd86640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7980acf70 0x7fb798148bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:19.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.928+0000 7fb79cd86640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb798146640 0x7fb798149110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:19.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.928+0000 7fb79cd86640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb798151cd0 con 0x7fb7980acf70 2024-04-25T19:44:19.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.928+0000 7fb79cd86640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb798151b50 con 0x7fb798146640 2024-04-25T19:44:19.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.928+0000 7fb79cd86640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fb798151e50 con 0x7fb79806df40 2024-04-25T19:44:19.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.928+0000 7fb79659c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb79806df40 0x7fb798147eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:19.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.928+0000 7fb795d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7980acf70 0x7fb798148bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:19.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.929+0000 7fb79659c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb79806df40 0x7fb798147eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:35178/0 (socket says 172.21.15.29:35178) 2024-04-25T19:44:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.929+0000 7fb795d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7980acf70 0x7fb798148bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39808/0 (socket says 172.21.15.29:39808) 2024-04-25T19:44:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.929+0000 7fb79659c640 1 -- 172.21.15.29:0/2746864193 learned_addr learned my addr 172.21.15.29:0/2746864193 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.929+0000 7fb796d9d640 1 --2- 172.21.15.29:0/2746864193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb798146640 0x7fb798149110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.929+0000 7fb795d9b640 1 -- 172.21.15.29:0/2746864193 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb79806df40 msgr2=0x7fb798147eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.929+0000 7fb795d9b640 1 --2- 172.21.15.29:0/2746864193 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb79806df40 0x7fb798147eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.929+0000 7fb795d9b640 1 -- 172.21.15.29:0/2746864193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb798146640 msgr2=0x7fb798149110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.929+0000 7fb795d9b640 1 --2- 172.21.15.29:0/2746864193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb798146640 0x7fb798149110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.929+0000 7fb795d9b640 1 -- 172.21.15.29:0/2746864193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7980a2380 con 0x7fb7980acf70 2024-04-25T19:44:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.929+0000 7fb796d9d640 1 --2- 172.21.15.29:0/2746864193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb798146640 0x7fb798149110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:44:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.929+0000 7fb795d9b640 1 --2- 172.21.15.29:0/2746864193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7980acf70 0x7fb798148bd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb78c033000 tx=0x7fb78c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.929+0000 7fb786ffd640 1 -- 172.21.15.29:0/2746864193 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb78c034040 con 0x7fb7980acf70 2024-04-25T19:44:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.929+0000 7fb786ffd640 1 -- 172.21.15.29:0/2746864193 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb78c03aaa0 con 0x7fb7980acf70 2024-04-25T19:44:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.929+0000 7fb79cd86640 1 -- 172.21.15.29:0/2746864193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7981483f0 con 0x7fb7980acf70 2024-04-25T19:44:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.929+0000 7fb786ffd640 1 -- 172.21.15.29:0/2746864193 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb78c0443f0 con 0x7fb7980acf70 2024-04-25T19:44:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.929+0000 7fb79cd86640 1 -- 172.21.15.29:0/2746864193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7980a5200 con 0x7fb7980acf70 2024-04-25T19:44:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.930+0000 7fb79cd86640 1 -- 172.21.15.29:0/2746864193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7980507b0 con 0x7fb7980acf70 2024-04-25T19:44:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.932+0000 7fb786ffd640 1 -- 172.21.15.29:0/2746864193 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7fb78c04b060 con 0x7fb7980acf70 2024-04-25T19:44:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.932+0000 7fb786ffd640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.932+0000 7fb786ffd640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.932+0000 7fb786ffd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.932+0000 7fb786ffd640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.932+0000 7fb786ffd640 1 --2- 172.21.15.29:0/2746864193 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fb7780ac530 0x7fb7780aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.932+0000 7fb786ffd640 1 -- 172.21.15.29:0/2746864193 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7fb78c0fb350 con 0x7fb7980acf70 2024-04-25T19:44:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.933+0000 7fb79659c640 1 --2- 172.21.15.29:0/2746864193 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fb7780ac530 0x7fb7780aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.933+0000 7fb79659c640 1 --2- 172.21.15.29:0/2746864193 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fb7780ac530 0x7fb7780aea40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb788005c00 tx=0x7fb788009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:19.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:19.936+0000 7fb786ffd640 1 -- 172.21.15.29:0/2746864193 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb78c07fb80 con 0x7fb7980acf70 2024-04-25T19:44:20.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.120+0000 7fb79cd86640 1 -- 172.21.15.29:0/2746864193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb7980aa4d0 con 0x7fb7980acf70 2024-04-25T19:44:20.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.121+0000 7fb786ffd640 1 -- 172.21.15.29:0/2746864193 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7fb78c03f070 con 0x7fb7980acf70 2024-04-25T19:44:20.122 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-04-25T19:44:20.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.122+0000 7fb79cd86640 10 mgrc shutdown 2024-04-25T19:44:20.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.123+0000 7fb79cd86640 1 -- 172.21.15.29:0/2746864193 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fb7780ac530 msgr2=0x7fb7780aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:20.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.123+0000 7fb79cd86640 1 --2- 172.21.15.29:0/2746864193 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fb7780ac530 0x7fb7780aea40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb788005c00 tx=0x7fb788009040 comp rx=0 tx=0).stop 2024-04-25T19:44:20.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.123+0000 7fb79cd86640 1 -- 172.21.15.29:0/2746864193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7980acf70 msgr2=0x7fb798148bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:20.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.123+0000 7fb79cd86640 1 --2- 172.21.15.29:0/2746864193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7980acf70 0x7fb798148bd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb78c033000 tx=0x7fb78c0046c0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.123+0000 7fb79cd86640 1 -- 172.21.15.29:0/2746864193 shutdown_connections 2024-04-25T19:44:20.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.123+0000 7fb79cd86640 1 --2- 172.21.15.29:0/2746864193 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fb7780ac530 0x7fb7780aea40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.123+0000 7fb79cd86640 1 --2- 172.21.15.29:0/2746864193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb798146640 0x7fb798149110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.123+0000 7fb79cd86640 1 --2- 172.21.15.29:0/2746864193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7980acf70 0x7fb798148bd0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.123+0000 7fb79cd86640 1 --2- 172.21.15.29:0/2746864193 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb79806df40 0x7fb798147eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.123+0000 7fb79cd86640 1 -- 172.21.15.29:0/2746864193 >> 172.21.15.29:0/2746864193 conn(0x7fb7980ab710 msgr2=0x7fb7980aa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:20.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.123+0000 7fb79cd86640 1 -- 172.21.15.29:0/2746864193 shutdown_connections 2024-04-25T19:44:20.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.123+0000 7fb79cd86640 1 -- 172.21.15.29:0/2746864193 wait complete. 2024-04-25T19:44:20.135 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:44:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.246+0000 7effd838e640 1 Processor -- start 2024-04-25T19:44:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.246+0000 7effd838e640 1 -- start start 2024-04-25T19:44:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.247+0000 7effd838e640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7effd0144240 0x7effd0146c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.247+0000 7effd838e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effd0144650 con 0x7effd0143e30 2024-04-25T19:44:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.247+0000 7effd838e640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effd00a5a90 con 0x7effd0147150 2024-04-25T19:44:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.247+0000 7effd838e640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7effd0147560 con 0x7effd0144240 2024-04-25T19:44:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.247+0000 7effd612a640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7effd0143e30 0x7effd00acaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52932/0 (socket says 172.21.15.29:52932) 2024-04-25T19:44:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.247+0000 7effd612a640 1 -- 172.21.15.29:0/4045092573 learned_addr learned my addr 172.21.15.29:0/4045092573 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.247+0000 7effd5929640 1 --2- 172.21.15.29:0/4045092573 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7effd0144240 0x7effd0146c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.248+0000 7effd5128640 1 -- 172.21.15.29:0/4045092573 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1394462355 0 0) 0x7effc8001110 con 0x7effd0143e30 2024-04-25T19:44:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.248+0000 7effd5128640 1 -- 172.21.15.29:0/4045092573 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7effb8003520 con 0x7effd0143e30 2024-04-25T19:44:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.248+0000 7effd5929640 1 -- 172.21.15.29:0/4045092573 >> v1:172.21.15.102:6789/0 conn(0x7effd0147150 legacy=0x7effd0151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.248+0000 7effd5929640 1 -- 172.21.15.29:0/4045092573 >> v1:172.21.15.29:6789/0 conn(0x7effd0143e30 legacy=0x7effd00acaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.248+0000 7effd5929640 1 -- 172.21.15.29:0/4045092573 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd0147790 con 0x7effd0144240 2024-04-25T19:44:20.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.249+0000 7effd5929640 1 --2- 172.21.15.29:0/4045092573 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7effd0144240 0x7effd0146c10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7effc4009800 tx=0x7effc402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=144cc331a6131748 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:20.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.249+0000 7effd5128640 1 -- 172.21.15.29:0/4045092573 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc4035020 con 0x7effd0144240 2024-04-25T19:44:20.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.249+0000 7effd5128640 1 -- 172.21.15.29:0/4045092573 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effc4031890 con 0x7effd0144240 2024-04-25T19:44:20.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.249+0000 7effd5128640 1 -- 172.21.15.29:0/4045092573 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc4031c60 con 0x7effd0144240 2024-04-25T19:44:20.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.249+0000 7effd838e640 1 -- 172.21.15.29:0/4045092573 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7effd0144240 msgr2=0x7effd0146c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:20.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.249+0000 7effd838e640 1 --2- 172.21.15.29:0/4045092573 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7effd0144240 0x7effd0146c10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7effc4009800 tx=0x7effc402b930 comp rx=0 tx=0).stop 2024-04-25T19:44:20.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.250+0000 7effd838e640 1 -- 172.21.15.29:0/4045092573 shutdown_connections 2024-04-25T19:44:20.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.250+0000 7effd838e640 1 --2- 172.21.15.29:0/4045092573 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7effd0144240 0x7effd0146c10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.250+0000 7effd838e640 1 -- 172.21.15.29:0/4045092573 >> 172.21.15.29:0/4045092573 conn(0x7effd00aafc0 msgr2=0x7effd00ab3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:20.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.250+0000 7effd838e640 1 -- 172.21.15.29:0/4045092573 shutdown_connections 2024-04-25T19:44:20.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.250+0000 7effd838e640 1 -- 172.21.15.29:0/4045092573 wait complete. 2024-04-25T19:44:20.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.251+0000 7effd838e640 1 Processor -- start 2024-04-25T19:44:20.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.251+0000 7effd838e640 1 -- start start 2024-04-25T19:44:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.252+0000 7effd838e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7effd0143e30 0x7effd01403e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.252+0000 7effd838e640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effd0144240 0x7effd014c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.252+0000 7effd838e640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7effd0147150 0x7effd014a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.252+0000 7effd838e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7effd00ad330 con 0x7effd0143e30 2024-04-25T19:44:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.252+0000 7effd838e640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7effd00ad1b0 con 0x7effd0144240 2024-04-25T19:44:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.252+0000 7effd838e640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7effd0152cb0 con 0x7effd0147150 2024-04-25T19:44:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.252+0000 7effd612a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7effd0143e30 0x7effd01403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.252+0000 7effd692b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7effd0147150 0x7effd014a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.252+0000 7effd692b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7effd0147150 0x7effd014a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:35188/0 (socket says 172.21.15.29:35188) 2024-04-25T19:44:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.252+0000 7effd692b640 1 -- 172.21.15.29:0/1328739245 learned_addr learned my addr 172.21.15.29:0/1328739245 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.252+0000 7effd5929640 1 --2- 172.21.15.29:0/1328739245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effd0144240 0x7effd014c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.252+0000 7effd692b640 1 -- 172.21.15.29:0/1328739245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effd0144240 msgr2=0x7effd014c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.252+0000 7effd692b640 1 --2- 172.21.15.29:0/1328739245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effd0144240 0x7effd014c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.252+0000 7effd692b640 1 -- 172.21.15.29:0/1328739245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7effd0143e30 msgr2=0x7effd01403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.252+0000 7effd692b640 1 --2- 172.21.15.29:0/1328739245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7effd0143e30 0x7effd01403e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.252+0000 7effd692b640 1 -- 172.21.15.29:0/1328739245 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd01365e0 con 0x7effd0147150 2024-04-25T19:44:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.252+0000 7effd612a640 1 --2- 172.21.15.29:0/1328739245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7effd0143e30 0x7effd01403e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.253+0000 7effd692b640 1 --2- 172.21.15.29:0/1328739245 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7effd0147150 0x7effd014a570 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7effc000c680 tx=0x7effc000cb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.253+0000 7effceffd640 1 -- 172.21.15.29:0/1328739245 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc0003b70 con 0x7effd0147150 2024-04-25T19:44:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.253+0000 7effceffd640 1 -- 172.21.15.29:0/1328739245 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effc0003d10 con 0x7effd0147150 2024-04-25T19:44:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.253+0000 7effceffd640 1 -- 172.21.15.29:0/1328739245 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc0005310 con 0x7effd0147150 2024-04-25T19:44:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.253+0000 7effd838e640 1 -- 172.21.15.29:0/1328739245 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effd0141120 con 0x7effd0147150 2024-04-25T19:44:20.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.253+0000 7effd838e640 1 -- 172.21.15.29:0/1328739245 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7effd0141630 con 0x7effd0147150 2024-04-25T19:44:20.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.254+0000 7effd838e640 1 -- 172.21.15.29:0/1328739245 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effd00507b0 con 0x7effd0147150 2024-04-25T19:44:20.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.255+0000 7effceffd640 1 -- 172.21.15.29:0/1328739245 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7effc00054b0 con 0x7effd0147150 2024-04-25T19:44:20.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.255+0000 7effceffd640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:20.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.256+0000 7effceffd640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:20.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.256+0000 7effceffd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:20.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.256+0000 7effceffd640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:20.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.256+0000 7effceffd640 1 --2- 172.21.15.29:0/1328739245 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7effbc0ac530 0x7effbc0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:20.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.256+0000 7effceffd640 1 -- 172.21.15.29:0/1328739245 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7effc00d9c30 con 0x7effd0147150 2024-04-25T19:44:20.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.256+0000 7effd612a640 1 --2- 172.21.15.29:0/1328739245 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7effbc0ac530 0x7effbc0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:20.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.257+0000 7effd612a640 1 --2- 172.21.15.29:0/1328739245 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7effbc0ac530 0x7effbc0aea40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7effc80012b0 tx=0x7effc8007b70 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:20.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.259+0000 7effceffd640 1 -- 172.21.15.29:0/1328739245 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7effc00a48a0 con 0x7effd0147150 2024-04-25T19:44:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.448+0000 7effd838e640 1 -- 172.21.15.29:0/1328739245 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7effd00acf40 con 0x7effd0147150 2024-04-25T19:44:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.449+0000 7effceffd640 1 -- 172.21.15.29:0/1328739245 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7effc00a41a0 con 0x7effd0147150 2024-04-25T19:44:20.450 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-04-25T19:44:20.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.451+0000 7effd838e640 10 mgrc shutdown 2024-04-25T19:44:20.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.451+0000 7effd838e640 1 -- 172.21.15.29:0/1328739245 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7effbc0ac530 msgr2=0x7effbc0aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:20.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.451+0000 7effd838e640 1 --2- 172.21.15.29:0/1328739245 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7effbc0ac530 0x7effbc0aea40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7effc80012b0 tx=0x7effc8007b70 comp rx=0 tx=0).stop 2024-04-25T19:44:20.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.451+0000 7effd838e640 1 -- 172.21.15.29:0/1328739245 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7effd0147150 msgr2=0x7effd014a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:20.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.451+0000 7effd838e640 1 --2- 172.21.15.29:0/1328739245 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7effd0147150 0x7effd014a570 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7effc000c680 tx=0x7effc000cb40 comp rx=0 tx=0).stop 2024-04-25T19:44:20.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.451+0000 7effd838e640 1 -- 172.21.15.29:0/1328739245 shutdown_connections 2024-04-25T19:44:20.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.451+0000 7effd838e640 1 --2- 172.21.15.29:0/1328739245 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7effbc0ac530 0x7effbc0aea40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.451+0000 7effd838e640 1 --2- 172.21.15.29:0/1328739245 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7effd0147150 0x7effd014a570 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.451+0000 7effd838e640 1 --2- 172.21.15.29:0/1328739245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effd0144240 0x7effd014c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.451+0000 7effd838e640 1 --2- 172.21.15.29:0/1328739245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7effd0143e30 0x7effd01403e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.451+0000 7effd838e640 1 -- 172.21.15.29:0/1328739245 >> 172.21.15.29:0/1328739245 conn(0x7effd00aafc0 msgr2=0x7effd00ba390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:20.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.451+0000 7effd838e640 1 -- 172.21.15.29:0/1328739245 shutdown_connections 2024-04-25T19:44:20.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.451+0000 7effd838e640 1 -- 172.21.15.29:0/1328739245 wait complete. 2024-04-25T19:44:20.463 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:44:20.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.573+0000 7fd159d22640 1 Processor -- start 2024-04-25T19:44:20.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.573+0000 7fd159d22640 1 -- start start 2024-04-25T19:44:20.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.574+0000 7fd159d22640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd1540a5c50 0x7fd1540ab9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:20.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.574+0000 7fd159d22640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1540ada60 con 0x7fd1540aa9a0 2024-04-25T19:44:20.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.574+0000 7fd159d22640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd15414a260 con 0x7fd15414aad0 2024-04-25T19:44:20.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.574+0000 7fd159d22640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fd1540aadb0 con 0x7fd1540a5c50 2024-04-25T19:44:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.574+0000 7fd1537fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd1540a5c50 0x7fd1540ab9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.574+0000 7fd152ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd1540aa9a0 0x7fd1540ad2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52938/0 (socket says 172.21.15.29:52938) 2024-04-25T19:44:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.574+0000 7fd152ffd640 1 -- 172.21.15.29:0/2375142466 learned_addr learned my addr 172.21.15.29:0/2375142466 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.574+0000 7fd1527fc640 1 -- 172.21.15.29:0/2375142466 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3444462392 0 0) 0x7fd1540ada60 con 0x7fd1540aa9a0 2024-04-25T19:44:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.574+0000 7fd1537fe640 1 -- 172.21.15.29:0/2375142466 >> v1:172.21.15.102:6789/0 conn(0x7fd15414aad0 legacy=0x7fd154151550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.574+0000 7fd1537fe640 1 -- 172.21.15.29:0/2375142466 >> v1:172.21.15.29:6789/0 conn(0x7fd1540aa9a0 legacy=0x7fd1540ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.574+0000 7fd1537fe640 1 -- 172.21.15.29:0/2375142466 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd15414aee0 con 0x7fd1540a5c50 2024-04-25T19:44:20.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.575+0000 7fd1537fe640 1 --2- 172.21.15.29:0/2375142466 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd1540a5c50 0x7fd1540ab9d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd1440097d0 tx=0x7fd14402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b0eb1595701745f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:20.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.575+0000 7fd1527fc640 1 -- 172.21.15.29:0/2375142466 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd144035020 con 0x7fd1540a5c50 2024-04-25T19:44:20.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.575+0000 7fd1527fc640 1 -- 172.21.15.29:0/2375142466 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd144031800 con 0x7fd1540a5c50 2024-04-25T19:44:20.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.575+0000 7fd1527fc640 1 -- 172.21.15.29:0/2375142466 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd144031bd0 con 0x7fd1540a5c50 2024-04-25T19:44:20.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.576+0000 7fd159d22640 1 -- 172.21.15.29:0/2375142466 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd1540a5c50 msgr2=0x7fd1540ab9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:20.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.576+0000 7fd159d22640 1 --2- 172.21.15.29:0/2375142466 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd1540a5c50 0x7fd1540ab9d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd1440097d0 tx=0x7fd14402b8a0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.576+0000 7fd159d22640 1 -- 172.21.15.29:0/2375142466 shutdown_connections 2024-04-25T19:44:20.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.576+0000 7fd159d22640 1 --2- 172.21.15.29:0/2375142466 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd1540a5c50 0x7fd1540ab9d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.576+0000 7fd159d22640 1 -- 172.21.15.29:0/2375142466 >> 172.21.15.29:0/2375142466 conn(0x7fd1540bc5c0 msgr2=0x7fd1540ab0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:20.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.576+0000 7fd159d22640 1 -- 172.21.15.29:0/2375142466 shutdown_connections 2024-04-25T19:44:20.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.576+0000 7fd159d22640 1 -- 172.21.15.29:0/2375142466 wait complete. 2024-04-25T19:44:20.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.577+0000 7fd159d22640 1 Processor -- start 2024-04-25T19:44:20.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.578+0000 7fd159d22640 1 -- start start 2024-04-25T19:44:20.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd159d22640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1540a5c50 0x7fd154149010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:20.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd159d22640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd1540aa9a0 0x7fd15414b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:20.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd159d22640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd15414aad0 0x7fd154143110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:20.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd159d22640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd154153000 con 0x7fd1540a5c50 2024-04-25T19:44:20.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd159d22640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd154152e80 con 0x7fd15414aad0 2024-04-25T19:44:20.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd159d22640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fd154153180 con 0x7fd1540aa9a0 2024-04-25T19:44:20.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd152ffd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd1540aa9a0 0x7fd15414b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:20.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd1537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1540a5c50 0x7fd154149010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:20.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd1537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1540a5c50 0x7fd154149010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39820/0 (socket says 172.21.15.29:39820) 2024-04-25T19:44:20.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd1537fe640 1 -- 172.21.15.29:0/1605037485 learned_addr learned my addr 172.21.15.29:0/1605037485 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:20.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd152ffd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd1540aa9a0 0x7fd15414b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:35212/0 (socket says 172.21.15.29:35212) 2024-04-25T19:44:20.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd153fff640 1 --2- 172.21.15.29:0/1605037485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd15414aad0 0x7fd154143110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:20.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd1537fe640 1 -- 172.21.15.29:0/1605037485 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd1540aa9a0 msgr2=0x7fd15414b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:20.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd1537fe640 1 --2- 172.21.15.29:0/1605037485 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd1540aa9a0 0x7fd15414b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd1537fe640 1 -- 172.21.15.29:0/1605037485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd15414aad0 msgr2=0x7fd154143110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:20.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd1537fe640 1 --2- 172.21.15.29:0/1605037485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd15414aad0 0x7fd154143110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd1537fe640 1 -- 172.21.15.29:0/1605037485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd154149b10 con 0x7fd1540a5c50 2024-04-25T19:44:20.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd1537fe640 1 --2- 172.21.15.29:0/1605037485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1540a5c50 0x7fd154149010 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd144009720 tx=0x7fd1440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:20.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd13bfff640 1 -- 172.21.15.29:0/1605037485 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14403ad60 con 0x7fd1540a5c50 2024-04-25T19:44:20.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd13bfff640 1 -- 172.21.15.29:0/1605037485 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1440443f0 con 0x7fd1540a5c50 2024-04-25T19:44:20.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd159d22640 1 -- 172.21.15.29:0/1605037485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd15414bcd0 con 0x7fd1540a5c50 2024-04-25T19:44:20.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd13bfff640 1 -- 172.21.15.29:0/1605037485 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1440447e0 con 0x7fd1540a5c50 2024-04-25T19:44:20.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.579+0000 7fd159d22640 1 -- 172.21.15.29:0/1605037485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd15413e5f0 con 0x7fd1540a5c50 2024-04-25T19:44:20.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.580+0000 7fd159d22640 1 -- 172.21.15.29:0/1605037485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd154146640 con 0x7fd1540a5c50 2024-04-25T19:44:20.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.582+0000 7fd13bfff640 1 -- 172.21.15.29:0/1605037485 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7fd14404b060 con 0x7fd1540a5c50 2024-04-25T19:44:20.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.582+0000 7fd13bfff640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:20.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.582+0000 7fd13bfff640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:20.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.582+0000 7fd13bfff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:20.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.582+0000 7fd13bfff640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:20.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.582+0000 7fd13bfff640 1 --2- 172.21.15.29:0/1605037485 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fd1400ac530 0x7fd1400aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:20.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.582+0000 7fd13bfff640 1 -- 172.21.15.29:0/1605037485 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7fd1440fb540 con 0x7fd1540a5c50 2024-04-25T19:44:20.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.583+0000 7fd152ffd640 1 --2- 172.21.15.29:0/1605037485 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fd1400ac530 0x7fd1400aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:20.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.583+0000 7fd152ffd640 1 --2- 172.21.15.29:0/1605037485 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fd1400ac530 0x7fd1400aea40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd154144600 tx=0x7fd148007de0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:20.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.586+0000 7fd13bfff640 1 -- 172.21.15.29:0/1605037485 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd14407fb80 con 0x7fd1540a5c50 2024-04-25T19:44:20.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.806+0000 7fd159d22640 1 -- 172.21.15.29:0/1605037485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1540507b0 con 0x7fd1540a5c50 2024-04-25T19:44:20.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.807+0000 7fd13bfff640 1 -- 172.21.15.29:0/1605037485 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+4351 (secure 0 0 0) 0x7fd14403f070 con 0x7fd1540a5c50 2024-04-25T19:44:20.808 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:20.808 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6809/3582557422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3582557422},{"type":"v1","addr":"172.21.15.29:6809","nonce":3582557422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4234,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/1228868047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1228868047},{"type":"v1","addr":"172.21.15.102:6827","nonce":1228868047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:44:09.410562+0000","modified":"2024-04-25T19:44:12.425418+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6811/844243878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":844243878},{"type":"v1","addr":"172.21.15.29:6811","nonce":844243878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":1}]} 2024-04-25T19:44:20.808 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2024-04-25T19:44:20.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.808+0000 7fd159d22640 10 mgrc shutdown 2024-04-25T19:44:20.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.808+0000 7fd159d22640 1 -- 172.21.15.29:0/1605037485 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fd1400ac530 msgr2=0x7fd1400aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:20.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.808+0000 7fd159d22640 1 --2- 172.21.15.29:0/1605037485 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fd1400ac530 0x7fd1400aea40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd154144600 tx=0x7fd148007de0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.808+0000 7fd159d22640 1 -- 172.21.15.29:0/1605037485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1540a5c50 msgr2=0x7fd154149010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:20.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.808+0000 7fd159d22640 1 --2- 172.21.15.29:0/1605037485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1540a5c50 0x7fd154149010 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd144009720 tx=0x7fd1440046c0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.809+0000 7fd159d22640 1 -- 172.21.15.29:0/1605037485 shutdown_connections 2024-04-25T19:44:20.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.809+0000 7fd159d22640 1 --2- 172.21.15.29:0/1605037485 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fd1400ac530 0x7fd1400aea40 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.809+0000 7fd159d22640 1 --2- 172.21.15.29:0/1605037485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd15414aad0 0x7fd154143110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.809+0000 7fd159d22640 1 --2- 172.21.15.29:0/1605037485 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd1540aa9a0 0x7fd15414b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.809+0000 7fd159d22640 1 --2- 172.21.15.29:0/1605037485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1540a5c50 0x7fd154149010 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.809+0000 7fd159d22640 1 -- 172.21.15.29:0/1605037485 >> 172.21.15.29:0/1605037485 conn(0x7fd1540bc5c0 msgr2=0x7fd1540b9e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:20.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.809+0000 7fd159d22640 1 -- 172.21.15.29:0/1605037485 shutdown_connections 2024-04-25T19:44:20.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.809+0000 7fd159d22640 1 -- 172.21.15.29:0/1605037485 wait complete. 2024-04-25T19:44:20.821 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:44:20.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.931+0000 7f67244cd640 1 Processor -- start 2024-04-25T19:44:20.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.931+0000 7f67244cd640 1 -- start start 2024-04-25T19:44:20.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.932+0000 7f67244cd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f671c0b70f0 0x7f671c0b7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:20.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.932+0000 7f67244cd640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f671c136f40 0x7f671c137350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.932+0000 7f67244cd640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f671c0ba510 con 0x7f671c0ba100 2024-04-25T19:44:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.932+0000 7f67244cd640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f671c137e90 con 0x7f671c136f40 2024-04-25T19:44:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.932+0000 7f67244cd640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f671c137bd0 con 0x7f671c0b70f0 2024-04-25T19:44:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.932+0000 7f6721a68640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f671c0b70f0 0x7f671c0b7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.932+0000 7f6722269640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f671c0ba100 0x7f671c0addb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52954/0 (socket says 172.21.15.29:52954) 2024-04-25T19:44:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.932+0000 7f6722269640 1 -- 172.21.15.29:0/2880531748 learned_addr learned my addr 172.21.15.29:0/2880531748 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.932+0000 7f6722a6a640 1 --2- 172.21.15.29:0/2880531748 >> v2:172.21.15.102:3300/0 conn(0x7f671c136f40 0x7f671c137350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.932+0000 7f6721267640 1 -- 172.21.15.29:0/2880531748 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 569988833 0 0) 0x7f6714001490 con 0x7f671c0ba100 2024-04-25T19:44:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.932+0000 7f6721a68640 1 -- 172.21.15.29:0/2880531748 >> v2:172.21.15.102:3300/0 conn(0x7f671c136f40 msgr2=0x7f671c137350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.932+0000 7f6721a68640 1 --2- 172.21.15.29:0/2880531748 >> v2:172.21.15.102:3300/0 conn(0x7f671c136f40 0x7f671c137350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.932+0000 7f6721a68640 1 -- 172.21.15.29:0/2880531748 >> v1:172.21.15.29:6789/0 conn(0x7f671c0ba100 legacy=0x7f671c0addb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.932+0000 7f6721a68640 1 -- 172.21.15.29:0/2880531748 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f671c137890 con 0x7f671c0b70f0 2024-04-25T19:44:20.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.933+0000 7f6721a68640 1 --2- 172.21.15.29:0/2880531748 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f671c0b70f0 0x7f671c0b7500 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f67100097a0 tx=0x7f671002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=feff83a3f63cd11b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:20.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.933+0000 7f6721267640 1 -- 172.21.15.29:0/2880531748 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6710035020 con 0x7f671c0b70f0 2024-04-25T19:44:20.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.933+0000 7f6721267640 1 -- 172.21.15.29:0/2880531748 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6710031830 con 0x7f671c0b70f0 2024-04-25T19:44:20.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.933+0000 7f6721267640 1 -- 172.21.15.29:0/2880531748 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6710031c00 con 0x7f671c0b70f0 2024-04-25T19:44:20.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.934+0000 7f67244cd640 1 -- 172.21.15.29:0/2880531748 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f671c0b70f0 msgr2=0x7f671c0b7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:20.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.934+0000 7f67244cd640 1 --2- 172.21.15.29:0/2880531748 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f671c0b70f0 0x7f671c0b7500 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f67100097a0 tx=0x7f671002b8d0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.934+0000 7f67244cd640 1 -- 172.21.15.29:0/2880531748 shutdown_connections 2024-04-25T19:44:20.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.934+0000 7f67244cd640 1 --2- 172.21.15.29:0/2880531748 >> v2:172.21.15.102:3300/0 conn(0x7f671c136f40 0x7f671c137350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.934+0000 7f67244cd640 1 --2- 172.21.15.29:0/2880531748 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f671c0b70f0 0x7f671c0b7500 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.934+0000 7f67244cd640 1 -- 172.21.15.29:0/2880531748 >> 172.21.15.29:0/2880531748 conn(0x7f671c072b90 msgr2=0x7f671c0a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:20.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.934+0000 7f67244cd640 1 -- 172.21.15.29:0/2880531748 shutdown_connections 2024-04-25T19:44:20.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.934+0000 7f67244cd640 1 -- 172.21.15.29:0/2880531748 wait complete. 2024-04-25T19:44:20.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.935+0000 7f67244cd640 1 Processor -- start 2024-04-25T19:44:20.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.935+0000 7f67244cd640 1 -- start start 2024-04-25T19:44:20.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.936+0000 7f67244cd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f671c0b70f0 0x7f671c1313b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:20.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.936+0000 7f67244cd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f671c0ba100 0x7f671c135640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:20.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.936+0000 7f67244cd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f671c136f40 0x7f671c1e09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:20.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.936+0000 7f67244cd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f671c14f5c0 con 0x7f671c136f40 2024-04-25T19:44:20.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.936+0000 7f67244cd640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f671c14f440 con 0x7f671c0ba100 2024-04-25T19:44:20.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.936+0000 7f67244cd640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f671c14f740 con 0x7f671c0b70f0 2024-04-25T19:44:20.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.936+0000 7f6722269640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f671c0b70f0 0x7f671c1313b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:20.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.936+0000 7f6722a6a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f671c136f40 0x7f671c1e09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:20.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.937+0000 7f6722269640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f671c0b70f0 0x7f671c1313b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:35230/0 (socket says 172.21.15.29:35230) 2024-04-25T19:44:20.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.937+0000 7f6722269640 1 -- 172.21.15.29:0/2066067720 learned_addr learned my addr 172.21.15.29:0/2066067720 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:20.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.937+0000 7f6721a68640 1 --2- 172.21.15.29:0/2066067720 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f671c0ba100 0x7f671c135640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:20.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.937+0000 7f6722a6a640 1 -- 172.21.15.29:0/2066067720 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f671c0b70f0 msgr2=0x7f671c1313b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:20.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.937+0000 7f6722a6a640 1 --2- 172.21.15.29:0/2066067720 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f671c0b70f0 0x7f671c1313b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.937+0000 7f6722a6a640 1 -- 172.21.15.29:0/2066067720 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f671c0ba100 msgr2=0x7f671c135640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:20.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.937+0000 7f6722a6a640 1 --2- 172.21.15.29:0/2066067720 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f671c0ba100 0x7f671c135640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:20.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.937+0000 7f6722a6a640 1 -- 172.21.15.29:0/2066067720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f671c131b50 con 0x7f671c136f40 2024-04-25T19:44:20.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.937+0000 7f6722269640 1 --2- 172.21.15.29:0/2066067720 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f671c0b70f0 0x7f671c1313b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:20.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.937+0000 7f6722a6a640 1 --2- 172.21.15.29:0/2066067720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f671c136f40 0x7f671c1e09a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f671c0aa910 tx=0x7f670c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:20.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.937+0000 7f671affd640 1 -- 172.21.15.29:0/2066067720 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c015070 con 0x7f671c136f40 2024-04-25T19:44:20.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.937+0000 7f671affd640 1 -- 172.21.15.29:0/2066067720 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f670c004260 con 0x7f671c136f40 2024-04-25T19:44:20.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.937+0000 7f671affd640 1 -- 172.21.15.29:0/2066067720 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c003610 con 0x7f671c136f40 2024-04-25T19:44:20.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.937+0000 7f67244cd640 1 -- 172.21.15.29:0/2066067720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f671c131da0 con 0x7f671c136f40 2024-04-25T19:44:20.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.937+0000 7f67244cd640 1 -- 172.21.15.29:0/2066067720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f671c0b6a70 con 0x7f671c136f40 2024-04-25T19:44:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.938+0000 7f67244cd640 1 -- 172.21.15.29:0/2066067720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f671c0507b0 con 0x7f671c136f40 2024-04-25T19:44:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.940+0000 7f671affd640 1 -- 172.21.15.29:0/2066067720 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f670c0037d0 con 0x7f671c136f40 2024-04-25T19:44:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.940+0000 7f671affd640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.940+0000 7f671affd640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.940+0000 7f671affd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.940+0000 7f671affd640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:20.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.940+0000 7f671affd640 1 --2- 172.21.15.29:0/2066067720 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f67080ac4b0 0x7f67080ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:20.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.940+0000 7f671affd640 1 -- 172.21.15.29:0/2066067720 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f670c0d8f90 con 0x7f671c136f40 2024-04-25T19:44:20.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.940+0000 7f6722269640 1 --2- 172.21.15.29:0/2066067720 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f67080ac4b0 0x7f67080ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:20.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.941+0000 7f6722269640 1 --2- 172.21.15.29:0/2066067720 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f67080ac4b0 0x7f67080ae9c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f67140012e0 tx=0x7f6714001700 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:20.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:20.945+0000 7f671affd640 1 -- 172.21.15.29:0/2066067720 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f671c131b50 con 0x7f671c136f40 2024-04-25T19:44:21.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.133+0000 7f67244cd640 1 -- 172.21.15.29:0/2066067720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f671c06e890 con 0x7f671c136f40 2024-04-25T19:44:21.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.134+0000 7f671affd640 1 -- 172.21.15.29:0/2066067720 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+13512 (secure 0 0 0) 0x7f670c0a35b0 con 0x7f671c136f40 2024-04-25T19:44:21.135 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:21.136 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"1b3c39b5-e787-409e-b0bd-f4f491c89d72","created":"2024-04-25T19:43:47.854697+0000","modified":"2024-04-25T19:44:11.035019+0000","last_up_change":"2024-04-25T19:43:57.853237+0000","last_in_change":"2024-04-25T19:43:49.022776+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:43:58.096700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T19:44:03.617011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:44:07.890493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2024-04-25T19:44:08.355570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d805223-ee84-494f-bd31-59091906aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6801","nonce":695704147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6803","nonce":695704147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6807","nonce":695704147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6805","nonce":695704147}]},"public_addr":"172.21.15.29:6801/695704147","cluster_addr":"172.21.15.29:6803/695704147","heartbeat_back_addr":"172.21.15.29:6807/695704147","heartbeat_front_addr":"172.21.15.29:6805/695704147","state":["exists","up"]},{"osd":1,"uuid":"607c5038-38af-440f-8371-e71758549920","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6814","nonce":3135821452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6817","nonce":3135821452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6823","nonce":3135821452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6821","nonce":3135821452}]},"public_addr":"172.21.15.102:6814/3135821452","cluster_addr":"172.21.15.102:6817/3135821452","heartbeat_back_addr":"172.21.15.102:6823/3135821452","heartbeat_front_addr":"172.21.15.102:6821/3135821452","state":["exists","up"]},{"osd":2,"uuid":"6859ae76-54bb-4c55-b0dc-5e2735c0d31c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6801","nonce":2071844144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6803","nonce":2071844144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6811","nonce":2071844144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6806","nonce":2071844144}]},"public_addr":"172.21.15.102:6801/2071844144","cluster_addr":"172.21.15.102:6803/2071844144","heartbeat_back_addr":"172.21.15.102:6811/2071844144","heartbeat_front_addr":"172.21.15.102:6806/2071844144","state":["exists","up"]},{"osd":3,"uuid":"a98b7b88-5206-4f22-84a6-78212e0d5f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6805","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6807","nonce":339944984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6810","nonce":339944984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6820","nonce":339944984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6815","nonce":339944984}]},"public_addr":"172.21.15.102:6807/339944984","cluster_addr":"172.21.15.102:6810/339944984","heartbeat_back_addr":"172.21.15.102:6820/339944984","heartbeat_front_addr":"172.21.15.102:6815/339944984","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.121529+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.201155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.264387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.198713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:44:21.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.137+0000 7f67244cd640 10 mgrc shutdown 2024-04-25T19:44:21.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.137+0000 7f67244cd640 1 -- 172.21.15.29:0/2066067720 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f67080ac4b0 msgr2=0x7f67080ae9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:21.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.137+0000 7f67244cd640 1 --2- 172.21.15.29:0/2066067720 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f67080ac4b0 0x7f67080ae9c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f67140012e0 tx=0x7f6714001700 comp rx=0 tx=0).stop 2024-04-25T19:44:21.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.137+0000 7f67244cd640 1 -- 172.21.15.29:0/2066067720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f671c136f40 msgr2=0x7f671c1e09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:21.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.137+0000 7f67244cd640 1 --2- 172.21.15.29:0/2066067720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f671c136f40 0x7f671c1e09a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f671c0aa910 tx=0x7f670c00c950 comp rx=0 tx=0).stop 2024-04-25T19:44:21.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.137+0000 7f67244cd640 1 -- 172.21.15.29:0/2066067720 shutdown_connections 2024-04-25T19:44:21.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.137+0000 7f67244cd640 1 --2- 172.21.15.29:0/2066067720 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f67080ac4b0 0x7f67080ae9c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.137+0000 7f67244cd640 1 --2- 172.21.15.29:0/2066067720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f671c136f40 0x7f671c1e09a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.137+0000 7f67244cd640 1 --2- 172.21.15.29:0/2066067720 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f671c0ba100 0x7f671c135640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.137+0000 7f67244cd640 1 --2- 172.21.15.29:0/2066067720 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f671c0b70f0 0x7f671c1313b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.137+0000 7f67244cd640 1 -- 172.21.15.29:0/2066067720 >> 172.21.15.29:0/2066067720 conn(0x7f671c072b90 msgr2=0x7f671c0ba6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:21.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.137+0000 7f67244cd640 1 -- 172.21.15.29:0/2066067720 shutdown_connections 2024-04-25T19:44:21.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.137+0000 7f67244cd640 1 -- 172.21.15.29:0/2066067720 wait complete. 2024-04-25T19:44:21.149 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:43:58.096700+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '13', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4, 'score_stable': 4, 'optimal_score': 0.5, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-25T19:44:03.617011+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T19:44:07.890493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T19:44:08.355570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T19:44:21.149 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:44:21.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.262+0000 7f5f10b9f640 1 Processor -- start 2024-04-25T19:44:21.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.262+0000 7f5f10b9f640 1 -- start start 2024-04-25T19:44:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.263+0000 7f5f10b9f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5f0c0aaa70 0x7f5f0c06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.263+0000 7f5f10b9f640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5f0c143fd0 0x7f5f0c0a96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.263+0000 7f5f10b9f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f0c0a21c0 con 0x7f5f0c1443e0 2024-04-25T19:44:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.263+0000 7f5f10b9f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f5f0c0aa280 con 0x7f5f0c0aaa70 2024-04-25T19:44:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.263+0000 7f5f10b9f640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f5f0c1437d0 con 0x7f5f0c143fd0 2024-04-25T19:44:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.263+0000 7f5f0b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5f0c0aaa70 0x7f5f0c06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.263+0000 7f5f0affd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5f0c143fd0 0x7f5f0c0a96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.263+0000 7f5f0bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5f0c1443e0 0x7f5f0c146c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52960/0 (socket says 172.21.15.29:52960) 2024-04-25T19:44:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.263+0000 7f5f0bfff640 1 -- 172.21.15.29:0/3651431386 learned_addr learned my addr 172.21.15.29:0/3651431386 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.263+0000 7f5f0b7fe640 1 -- 172.21.15.29:0/3651431386 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5f0c143fd0 msgr2=0x7f5f0c0a96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.263+0000 7f5f0b7fe640 1 --2- 172.21.15.29:0/3651431386 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5f0c143fd0 0x7f5f0c0a96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.263+0000 7f5f0b7fe640 1 -- 172.21.15.29:0/3651431386 >> v1:172.21.15.29:6789/0 conn(0x7f5f0c1443e0 legacy=0x7f5f0c146c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:21.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.263+0000 7f5f0b7fe640 1 -- 172.21.15.29:0/3651431386 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f0c0a9bf0 con 0x7f5f0c0aaa70 2024-04-25T19:44:21.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.263+0000 7f5f0a7fc640 1 -- 172.21.15.29:0/3651431386 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 165619210 0 0) 0x7f5f0c0a21c0 con 0x7f5f0c1443e0 2024-04-25T19:44:21.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.263+0000 7f5f0affd640 1 --2- 172.21.15.29:0/3651431386 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5f0c143fd0 0x7f5f0c0a96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:21.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.264+0000 7f5f0b7fe640 1 --2- 172.21.15.29:0/3651431386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f0c0aaa70 0x7f5f0c06ed10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5f000097a0 tx=0x7f5f0002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6ceae49baf15a29 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:21.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.264+0000 7f5f0a7fc640 1 -- 172.21.15.29:0/3651431386 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f00035020 con 0x7f5f0c0aaa70 2024-04-25T19:44:21.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.264+0000 7f5f0a7fc640 1 -- 172.21.15.29:0/3651431386 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f00031830 con 0x7f5f0c0aaa70 2024-04-25T19:44:21.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.264+0000 7f5f0a7fc640 1 -- 172.21.15.29:0/3651431386 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f00031c00 con 0x7f5f0c0aaa70 2024-04-25T19:44:21.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.265+0000 7f5f10b9f640 1 -- 172.21.15.29:0/3651431386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f0c0aaa70 msgr2=0x7f5f0c06ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:21.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.265+0000 7f5f10b9f640 1 --2- 172.21.15.29:0/3651431386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f0c0aaa70 0x7f5f0c06ed10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5f000097a0 tx=0x7f5f0002b8d0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.265+0000 7f5f10b9f640 1 -- 172.21.15.29:0/3651431386 shutdown_connections 2024-04-25T19:44:21.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.265+0000 7f5f10b9f640 1 --2- 172.21.15.29:0/3651431386 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5f0c143fd0 0x7f5f0c0a96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.265+0000 7f5f10b9f640 1 --2- 172.21.15.29:0/3651431386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f0c0aaa70 0x7f5f0c06ed10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.265+0000 7f5f10b9f640 1 -- 172.21.15.29:0/3651431386 >> 172.21.15.29:0/3651431386 conn(0x7f5f0c06d6a0 msgr2=0x7f5f0c06dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:21.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.265+0000 7f5f10b9f640 1 -- 172.21.15.29:0/3651431386 shutdown_connections 2024-04-25T19:44:21.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.265+0000 7f5f10b9f640 1 -- 172.21.15.29:0/3651431386 wait complete. 2024-04-25T19:44:21.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.266+0000 7f5f10b9f640 1 Processor -- start 2024-04-25T19:44:21.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.267+0000 7f5f10b9f640 1 -- start start 2024-04-25T19:44:21.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.267+0000 7f5f10b9f640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f0c0aaa70 0x7f5f0c1cf080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:21.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5f10b9f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f0c143fd0 0x7f5f0c1dda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:21.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5f10b9f640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5f0c1443e0 0x7f5f0c06c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:21.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5f10b9f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c1474c0 con 0x7f5f0c143fd0 2024-04-25T19:44:21.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5f10b9f640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c147340 con 0x7f5f0c0aaa70 2024-04-25T19:44:21.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5f10b9f640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f5f0c147640 con 0x7f5f0c1443e0 2024-04-25T19:44:21.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5f0affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f0c143fd0 0x7f5f0c1dda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:21.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5f0bfff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5f0c1443e0 0x7f5f0c06c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:21.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5f0affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f0c143fd0 0x7f5f0c1dda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39848/0 (socket says 172.21.15.29:39848) 2024-04-25T19:44:21.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5f0affd640 1 -- 172.21.15.29:0/1770219904 learned_addr learned my addr 172.21.15.29:0/1770219904 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:21.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5f0b7fe640 1 --2- 172.21.15.29:0/1770219904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f0c0aaa70 0x7f5f0c1cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:21.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5f0bfff640 1 -- 172.21.15.29:0/1770219904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f0c0aaa70 msgr2=0x7f5f0c1cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:21.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5f0bfff640 1 --2- 172.21.15.29:0/1770219904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f0c0aaa70 0x7f5f0c1cf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5f0bfff640 1 -- 172.21.15.29:0/1770219904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f0c143fd0 msgr2=0x7f5f0c1dda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:21.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5f0bfff640 1 --2- 172.21.15.29:0/1770219904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f0c143fd0 0x7f5f0c1dda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5f0bfff640 1 -- 172.21.15.29:0/1770219904 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f0c0a48b0 con 0x7f5f0c1443e0 2024-04-25T19:44:21.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5f0affd640 1 --2- 172.21.15.29:0/1770219904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f0c143fd0 0x7f5f0c1dda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:21.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5f0bfff640 1 --2- 172.21.15.29:0/1770219904 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5f0c1443e0 0x7f5f0c06c590 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c0a7770 tx=0x7f5f0400cbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:21.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5ef3fff640 1 -- 172.21.15.29:0/1770219904 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f04018070 con 0x7f5f0c1443e0 2024-04-25T19:44:21.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5ef3fff640 1 -- 172.21.15.29:0/1770219904 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f04004d60 con 0x7f5f0c1443e0 2024-04-25T19:44:21.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5f10b9f640 1 -- 172.21.15.29:0/1770219904 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f0c1ddf80 con 0x7f5f0c1443e0 2024-04-25T19:44:21.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5ef3fff640 1 -- 172.21.15.29:0/1770219904 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f0400a850 con 0x7f5f0c1443e0 2024-04-25T19:44:21.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.268+0000 7f5f10b9f640 1 -- 172.21.15.29:0/1770219904 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f0c143020 con 0x7f5f0c1443e0 2024-04-25T19:44:21.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.269+0000 7f5f10b9f640 1 -- 172.21.15.29:0/1770219904 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f0c1de1b0 con 0x7f5f0c1443e0 2024-04-25T19:44:21.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.271+0000 7f5ef3fff640 1 -- 172.21.15.29:0/1770219904 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f5f0400a9f0 con 0x7f5f0c1443e0 2024-04-25T19:44:21.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.271+0000 7f5ef3fff640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:21.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.271+0000 7f5ef3fff640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:21.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.271+0000 7f5ef3fff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:21.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.271+0000 7f5ef3fff640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:21.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.271+0000 7f5ef3fff640 1 --2- 172.21.15.29:0/1770219904 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5ef80ac4b0 0x7f5ef80ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:21.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.272+0000 7f5ef3fff640 1 -- 172.21.15.29:0/1770219904 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f5f040d8d00 con 0x7f5f0c1443e0 2024-04-25T19:44:21.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.272+0000 7f5f0b7fe640 1 --2- 172.21.15.29:0/1770219904 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5ef80ac4b0 0x7f5ef80ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:21.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.272+0000 7f5f0b7fe640 1 --2- 172.21.15.29:0/1770219904 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5ef80ac4b0 0x7f5ef80ae9c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5f00036000 tx=0x7f5f00008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:21.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.275+0000 7f5ef3fff640 1 -- 172.21.15.29:0/1770219904 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5f0c1de1b0 con 0x7f5f0c1443e0 2024-04-25T19:44:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.457+0000 7f5f10b9f640 1 -- 172.21.15.29:0/1770219904 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5f0c050740 con 0x7f5f0c1443e0 2024-04-25T19:44:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.457+0000 7f5ef3fff640 1 -- 172.21.15.29:0/1770219904 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f5f04021020 con 0x7f5f0c1443e0 2024-04-25T19:44:21.458 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-04-25T19:44:21.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.459+0000 7f5f10b9f640 10 mgrc shutdown 2024-04-25T19:44:21.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.459+0000 7f5f10b9f640 1 -- 172.21.15.29:0/1770219904 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5ef80ac4b0 msgr2=0x7f5ef80ae9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:21.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.459+0000 7f5f10b9f640 1 --2- 172.21.15.29:0/1770219904 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5ef80ac4b0 0x7f5ef80ae9c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5f00036000 tx=0x7f5f00008030 comp rx=0 tx=0).stop 2024-04-25T19:44:21.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.459+0000 7f5f10b9f640 1 -- 172.21.15.29:0/1770219904 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5f0c1443e0 msgr2=0x7f5f0c06c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:21.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.459+0000 7f5f10b9f640 1 --2- 172.21.15.29:0/1770219904 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5f0c1443e0 0x7f5f0c06c590 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c0a7770 tx=0x7f5f0400cbe0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.459+0000 7f5f10b9f640 1 -- 172.21.15.29:0/1770219904 shutdown_connections 2024-04-25T19:44:21.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.459+0000 7f5f10b9f640 1 --2- 172.21.15.29:0/1770219904 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5ef80ac4b0 0x7f5ef80ae9c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.459+0000 7f5f10b9f640 1 --2- 172.21.15.29:0/1770219904 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5f0c1443e0 0x7f5f0c06c590 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.459+0000 7f5f10b9f640 1 --2- 172.21.15.29:0/1770219904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f0c143fd0 0x7f5f0c1dda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.459+0000 7f5f10b9f640 1 --2- 172.21.15.29:0/1770219904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f0c0aaa70 0x7f5f0c1cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.459+0000 7f5f10b9f640 1 -- 172.21.15.29:0/1770219904 >> 172.21.15.29:0/1770219904 conn(0x7f5f0c06d6a0 msgr2=0x7f5f0c06dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:21.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.459+0000 7f5f10b9f640 1 -- 172.21.15.29:0/1770219904 shutdown_connections 2024-04-25T19:44:21.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.459+0000 7f5f10b9f640 1 -- 172.21.15.29:0/1770219904 wait complete. 2024-04-25T19:44:21.474 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-04-25T19:44:21.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.585+0000 7f58e7858640 1 Processor -- start 2024-04-25T19:44:21.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.586+0000 7f58e7858640 1 -- start start 2024-04-25T19:44:21.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.587+0000 7f58e7858640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f58e00ae2a0 0x7f58e00bc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:21.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.587+0000 7f58e7858640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f58e00ad690 0x7f58e014aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:21.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.587+0000 7f58e7858640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58e00adaa0 con 0x7f58e00ad280 2024-04-25T19:44:21.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.587+0000 7f58e7858640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f58e00a5fb0 con 0x7f58e00ad690 2024-04-25T19:44:21.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.587+0000 7f58e7858640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f58e0147da0 con 0x7f58e00ae2a0 2024-04-25T19:44:21.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.587+0000 7f58e55f4640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f58e00ae2a0 0x7f58e00bc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:21.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.587+0000 7f58e5df5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f58e00ad690 0x7f58e014aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:21.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.587+0000 7f58e55f4640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f58e00ae2a0 0x7f58e00bc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:35260/0 (socket says 172.21.15.29:35260) 2024-04-25T19:44:21.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.587+0000 7f58e5df5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f58e00ad690 0x7f58e014aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39858/0 (socket says 172.21.15.29:39858) 2024-04-25T19:44:21.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.587+0000 7f58e55f4640 1 -- 172.21.15.29:0/2271763569 learned_addr learned my addr 172.21.15.29:0/2271763569 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:21.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.587+0000 7f58e55f4640 1 -- 172.21.15.29:0/2271763569 >> v2:172.21.15.29:3300/0 conn(0x7f58e00ad690 msgr2=0x7f58e014aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:21.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.587+0000 7f58e55f4640 1 --2- 172.21.15.29:0/2271763569 >> v2:172.21.15.29:3300/0 conn(0x7f58e00ad690 0x7f58e014aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.587+0000 7f58e55f4640 1 -- 172.21.15.29:0/2271763569 >> v1:172.21.15.102:6789/0 conn(0x7f58e00ad280 legacy=0x7f58e014a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:21.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.587+0000 7f58e55f4640 1 -- 172.21.15.29:0/2271763569 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58e0146ef0 con 0x7f58e00ae2a0 2024-04-25T19:44:21.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.588+0000 7f58e55f4640 1 --2- 172.21.15.29:0/2271763569 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f58e00ae2a0 0x7f58e00bc640 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f58d0009e50 tx=0x7f58d002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccb26e3ffa0200f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:21.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.588+0000 7f58dffff640 1 -- 172.21.15.29:0/2271763569 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d002c9e0 con 0x7f58e00ae2a0 2024-04-25T19:44:21.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.588+0000 7f58dffff640 1 -- 172.21.15.29:0/2271763569 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58d002cb80 con 0x7f58e00ae2a0 2024-04-25T19:44:21.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.588+0000 7f58dffff640 1 -- 172.21.15.29:0/2271763569 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d00343f0 con 0x7f58e00ae2a0 2024-04-25T19:44:21.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.589+0000 7f58e7858640 1 -- 172.21.15.29:0/2271763569 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f58e00ae2a0 msgr2=0x7f58e00bc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:21.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.589+0000 7f58e7858640 1 --2- 172.21.15.29:0/2271763569 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f58e00ae2a0 0x7f58e00bc640 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f58d0009e50 tx=0x7f58d002be40 comp rx=0 tx=0).stop 2024-04-25T19:44:21.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.589+0000 7f58e7858640 1 -- 172.21.15.29:0/2271763569 shutdown_connections 2024-04-25T19:44:21.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.589+0000 7f58e7858640 1 --2- 172.21.15.29:0/2271763569 >> v2:172.21.15.29:3300/0 conn(0x7f58e00ad690 0x7f58e014aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.589+0000 7f58e7858640 1 --2- 172.21.15.29:0/2271763569 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f58e00ae2a0 0x7f58e00bc640 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.589+0000 7f58e7858640 1 -- 172.21.15.29:0/2271763569 >> 172.21.15.29:0/2271763569 conn(0x7f58e00ab820 msgr2=0x7f58e00bbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:21.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.589+0000 7f58e7858640 1 -- 172.21.15.29:0/2271763569 shutdown_connections 2024-04-25T19:44:21.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.589+0000 7f58e7858640 1 -- 172.21.15.29:0/2271763569 wait complete. 2024-04-25T19:44:21.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.590+0000 7f58e7858640 1 Processor -- start 2024-04-25T19:44:21.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.590+0000 7f58e7858640 1 -- start start 2024-04-25T19:44:21.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.591+0000 7f58e7858640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58e00ad280 0x7f58e009e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:21.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.591+0000 7f58e7858640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58e00ad690 0x7f58e01dd110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:21.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.591+0000 7f58e7858640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f58e00ae2a0 0x7f58e0143de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:21.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.591+0000 7f58e7858640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f58e00bcb80 con 0x7f58e00ad690 2024-04-25T19:44:21.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.591+0000 7f58e7858640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f58e00ae6b0 con 0x7f58e00ad280 2024-04-25T19:44:21.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.591+0000 7f58e7858640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f58e00a5a00 con 0x7f58e00ae2a0 2024-04-25T19:44:21.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.591+0000 7f58e5df5640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f58e00ae2a0 0x7f58e0143de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:21.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.591+0000 7f58e4df3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58e00ad690 0x7f58e01dd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:21.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.591+0000 7f58e5df5640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f58e00ae2a0 0x7f58e0143de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:35274/0 (socket says 172.21.15.29:35274) 2024-04-25T19:44:21.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.591+0000 7f58e5df5640 1 -- 172.21.15.29:0/2038241973 learned_addr learned my addr 172.21.15.29:0/2038241973 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:21.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.592+0000 7f58e55f4640 1 --2- 172.21.15.29:0/2038241973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58e00ad280 0x7f58e009e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:21.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.592+0000 7f58e5df5640 1 -- 172.21.15.29:0/2038241973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58e00ad280 msgr2=0x7f58e009e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:21.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.592+0000 7f58e5df5640 1 --2- 172.21.15.29:0/2038241973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58e00ad280 0x7f58e009e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.592+0000 7f58e5df5640 1 -- 172.21.15.29:0/2038241973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58e00ad690 msgr2=0x7f58e01dd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:21.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.592+0000 7f58e5df5640 1 --2- 172.21.15.29:0/2038241973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58e00ad690 0x7f58e01dd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.592+0000 7f58e5df5640 1 -- 172.21.15.29:0/2038241973 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58e01dcbf0 con 0x7f58e00ae2a0 2024-04-25T19:44:21.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.592+0000 7f58e4df3640 1 --2- 172.21.15.29:0/2038241973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58e00ad690 0x7f58e01dd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:21.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.592+0000 7f58e5df5640 1 --2- 172.21.15.29:0/2038241973 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f58e00ae2a0 0x7f58e0143de0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f58d800e7f0 tx=0x7f58d800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:21.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.592+0000 7f58ddffb640 1 -- 172.21.15.29:0/2038241973 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d8017070 con 0x7f58e00ae2a0 2024-04-25T19:44:21.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.592+0000 7f58ddffb640 1 -- 172.21.15.29:0/2038241973 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58d800c730 con 0x7f58e00ae2a0 2024-04-25T19:44:21.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.592+0000 7f58e7858640 1 -- 172.21.15.29:0/2038241973 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58e009ea30 con 0x7f58e00ae2a0 2024-04-25T19:44:21.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.592+0000 7f58ddffb640 1 -- 172.21.15.29:0/2038241973 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d800cb20 con 0x7f58e00ae2a0 2024-04-25T19:44:21.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.592+0000 7f58e7858640 1 -- 172.21.15.29:0/2038241973 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f58e0144cf0 con 0x7f58e00ae2a0 2024-04-25T19:44:21.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.594+0000 7f58e7858640 1 -- 172.21.15.29:0/2038241973 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58e00507b0 con 0x7f58e00ae2a0 2024-04-25T19:44:21.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.595+0000 7f58ddffb640 1 -- 172.21.15.29:0/2038241973 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f58d801a070 con 0x7f58e00ae2a0 2024-04-25T19:44:21.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.595+0000 7f58ddffb640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:21.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.595+0000 7f58ddffb640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:21.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.595+0000 7f58ddffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:21.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.595+0000 7f58ddffb640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:21.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.595+0000 7f58ddffb640 1 --2- 172.21.15.29:0/2038241973 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f58cc0ac530 0x7f58cc0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:21.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.595+0000 7f58ddffb640 1 -- 172.21.15.29:0/2038241973 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f58d80d94d0 con 0x7f58e00ae2a0 2024-04-25T19:44:21.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.596+0000 7f58e55f4640 1 --2- 172.21.15.29:0/2038241973 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f58cc0ac530 0x7f58cc0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:21.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.596+0000 7f58e55f4640 1 --2- 172.21.15.29:0/2038241973 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f58cc0ac530 0x7f58cc0aea40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f58d0035000 tx=0x7f58d0030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:21.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.600+0000 7f58ddffb640 1 -- 172.21.15.29:0/2038241973 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f58d80a3af0 con 0x7f58e00ae2a0 2024-04-25T19:44:21.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.782+0000 7f58e7858640 1 -- 172.21.15.29:0/2038241973 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7f58e0145740 con 0x7f58e00ae2a0 2024-04-25T19:44:21.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.782+0000 7f58ddffb640 1 -- 172.21.15.29:0/2038241973 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) ==== 92+0+10 (secure 0 0 0) 0x7f58d80a7df0 con 0x7f58e00ae2a0 2024-04-25T19:44:21.784 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 8 2024-04-25T19:44:21.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.784+0000 7f58e7858640 10 mgrc shutdown 2024-04-25T19:44:21.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.784+0000 7f58e7858640 1 -- 172.21.15.29:0/2038241973 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f58cc0ac530 msgr2=0x7f58cc0aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:21.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.784+0000 7f58e7858640 1 --2- 172.21.15.29:0/2038241973 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f58cc0ac530 0x7f58cc0aea40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f58d0035000 tx=0x7f58d0030000 comp rx=0 tx=0).stop 2024-04-25T19:44:21.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.784+0000 7f58e7858640 1 -- 172.21.15.29:0/2038241973 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f58e00ae2a0 msgr2=0x7f58e0143de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:21.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.784+0000 7f58e7858640 1 --2- 172.21.15.29:0/2038241973 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f58e00ae2a0 0x7f58e0143de0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f58d800e7f0 tx=0x7f58d800ecb0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.784+0000 7f58e7858640 1 -- 172.21.15.29:0/2038241973 shutdown_connections 2024-04-25T19:44:21.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.784+0000 7f58e7858640 1 --2- 172.21.15.29:0/2038241973 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f58cc0ac530 0x7f58cc0aea40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.784+0000 7f58e7858640 1 --2- 172.21.15.29:0/2038241973 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f58e00ae2a0 0x7f58e0143de0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.784+0000 7f58e7858640 1 --2- 172.21.15.29:0/2038241973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58e00ad690 0x7f58e01dd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.784+0000 7f58e7858640 1 --2- 172.21.15.29:0/2038241973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58e00ad280 0x7f58e009e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.784+0000 7f58e7858640 1 -- 172.21.15.29:0/2038241973 >> 172.21.15.29:0/2038241973 conn(0x7f58e00ab820 msgr2=0x7f58e00a7470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:21.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.784+0000 7f58e7858640 1 -- 172.21.15.29:0/2038241973 shutdown_connections 2024-04-25T19:44:21.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.784+0000 7f58e7858640 1 -- 172.21.15.29:0/2038241973 wait complete. 2024-04-25T19:44:21.797 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:44:21.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.909+0000 7f25497c6640 1 Processor -- start 2024-04-25T19:44:21.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.909+0000 7f25497c6640 1 -- start start 2024-04-25T19:44:21.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.910+0000 7f25497c6640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f254406da90 0x7f25440a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:21.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.910+0000 7f25497c6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2544146c60 con 0x7f25441472d0 2024-04-25T19:44:21.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.910+0000 7f25497c6640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25440ad9d0 con 0x7f25440ad450 2024-04-25T19:44:21.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.910+0000 7f25497c6640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f25440a5c10 con 0x7f254406da90 2024-04-25T19:44:21.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.910+0000 7f253bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f25441472d0 0x7f25440acd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52962/0 (socket says 172.21.15.29:52962) 2024-04-25T19:44:21.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.910+0000 7f253bfff640 1 -- 172.21.15.29:0/3725595165 learned_addr learned my addr 172.21.15.29:0/3725595165 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:21.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.910+0000 7f2542ffd640 1 --2- 172.21.15.29:0/3725595165 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f254406da90 0x7f25440a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:21.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.910+0000 7f25427fc640 1 -- 172.21.15.29:0/3725595165 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1271908120 0 0) 0x7f2544146c60 con 0x7f25441472d0 2024-04-25T19:44:21.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.910+0000 7f25427fc640 1 -- 172.21.15.29:0/3725595165 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2530003520 con 0x7f25441472d0 2024-04-25T19:44:21.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.911+0000 7f2542ffd640 1 -- 172.21.15.29:0/3725595165 >> v1:172.21.15.102:6789/0 conn(0x7f25440ad450 legacy=0x7f25441513f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:21.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.911+0000 7f2542ffd640 1 -- 172.21.15.29:0/3725595165 >> v1:172.21.15.29:6789/0 conn(0x7f25441472d0 legacy=0x7f25440acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:21.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.911+0000 7f2542ffd640 1 -- 172.21.15.29:0/3725595165 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25441476e0 con 0x7f254406da90 2024-04-25T19:44:21.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.911+0000 7f2542ffd640 1 --2- 172.21.15.29:0/3725595165 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f254406da90 0x7f25440a56d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f25340097d0 tx=0x7f253402b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf58ff8ffa83f4cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:21.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.911+0000 7f25427fc640 1 -- 172.21.15.29:0/3725595165 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2534034070 con 0x7f254406da90 2024-04-25T19:44:21.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.912+0000 7f25427fc640 1 -- 172.21.15.29:0/3725595165 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2534030880 con 0x7f254406da90 2024-04-25T19:44:21.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.912+0000 7f25427fc640 1 -- 172.21.15.29:0/3725595165 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2534030c30 con 0x7f254406da90 2024-04-25T19:44:21.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.912+0000 7f25497c6640 1 -- 172.21.15.29:0/3725595165 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f254406da90 msgr2=0x7f25440a56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:21.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.912+0000 7f25497c6640 1 --2- 172.21.15.29:0/3725595165 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f254406da90 0x7f25440a56d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f25340097d0 tx=0x7f253402b9c0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.912+0000 7f25497c6640 1 -- 172.21.15.29:0/3725595165 shutdown_connections 2024-04-25T19:44:21.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.912+0000 7f25497c6640 1 --2- 172.21.15.29:0/3725595165 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f254406da90 0x7f25440a56d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.912+0000 7f25497c6640 1 -- 172.21.15.29:0/3725595165 >> 172.21.15.29:0/3725595165 conn(0x7f25440bc3b0 msgr2=0x7f25440bc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:21.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.912+0000 7f25497c6640 1 -- 172.21.15.29:0/3725595165 shutdown_connections 2024-04-25T19:44:21.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.912+0000 7f25497c6640 1 -- 172.21.15.29:0/3725595165 wait complete. 2024-04-25T19:44:21.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.914+0000 7f25497c6640 1 Processor -- start 2024-04-25T19:44:21.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.914+0000 7f25497c6640 1 -- start start 2024-04-25T19:44:21.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.915+0000 7f25497c6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f254406da90 0x7f25441401f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:21.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.915+0000 7f25497c6640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f25440ad450 0x7f2544145770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:21.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.915+0000 7f25497c6640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25441472d0 0x7f25441419b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:21.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.915+0000 7f25497c6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2544152c90 con 0x7f254406da90 2024-04-25T19:44:21.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.915+0000 7f25497c6640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2544152b10 con 0x7f25441472d0 2024-04-25T19:44:21.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.915+0000 7f25497c6640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f2544152e10 con 0x7f25440ad450 2024-04-25T19:44:21.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.915+0000 7f253bfff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f25440ad450 0x7f2544145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:21.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.915+0000 7f2542ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f254406da90 0x7f25441401f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:21.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.915+0000 7f253bfff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f25440ad450 0x7f2544145770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:35278/0 (socket says 172.21.15.29:35278) 2024-04-25T19:44:21.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.915+0000 7f253bfff640 1 -- 172.21.15.29:0/1468885579 learned_addr learned my addr 172.21.15.29:0/1468885579 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:21.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.915+0000 7f25437fe640 1 --2- 172.21.15.29:0/1468885579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25441472d0 0x7f25441419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:21.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.915+0000 7f253bfff640 1 -- 172.21.15.29:0/1468885579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25441472d0 msgr2=0x7f25441419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:21.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.915+0000 7f253bfff640 1 --2- 172.21.15.29:0/1468885579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25441472d0 0x7f25441419b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.915+0000 7f253bfff640 1 -- 172.21.15.29:0/1468885579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f254406da90 msgr2=0x7f25441401f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:21.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.915+0000 7f253bfff640 1 --2- 172.21.15.29:0/1468885579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f254406da90 0x7f25441401f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:21.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.915+0000 7f253bfff640 1 -- 172.21.15.29:0/1468885579 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2544142d00 con 0x7f25440ad450 2024-04-25T19:44:21.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.915+0000 7f25437fe640 1 --2- 172.21.15.29:0/1468885579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25441472d0 0x7f25441419b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:44:21.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.915+0000 7f253bfff640 1 --2- 172.21.15.29:0/1468885579 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f25440ad450 0x7f2544145770 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f252c00c410 tx=0x7f252c00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:21.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.915+0000 7f253b7fe640 1 -- 172.21.15.29:0/1468885579 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c017070 con 0x7f25440ad450 2024-04-25T19:44:21.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.916+0000 7f253b7fe640 1 -- 172.21.15.29:0/1468885579 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f252c005ce0 con 0x7f25440ad450 2024-04-25T19:44:21.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.916+0000 7f253b7fe640 1 -- 172.21.15.29:0/1468885579 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c0072f0 con 0x7f25440ad450 2024-04-25T19:44:21.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.916+0000 7f25497c6640 1 -- 172.21.15.29:0/1468885579 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2544144e70 con 0x7f25440ad450 2024-04-25T19:44:21.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.916+0000 7f25497c6640 1 -- 172.21.15.29:0/1468885579 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2544145cb0 con 0x7f25440ad450 2024-04-25T19:44:21.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.916+0000 7f25497c6640 1 -- 172.21.15.29:0/1468885579 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25440507b0 con 0x7f25440ad450 2024-04-25T19:44:21.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.918+0000 7f253b7fe640 1 -- 172.21.15.29:0/1468885579 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f252c01a070 con 0x7f25440ad450 2024-04-25T19:44:21.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.918+0000 7f253b7fe640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:21.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.919+0000 7f253b7fe640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:21.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.919+0000 7f253b7fe640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:21.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.919+0000 7f253b7fe640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:21.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.919+0000 7f253b7fe640 1 --2- 172.21.15.29:0/1468885579 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f25240ac530 0x7f25240aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:21.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.919+0000 7f253b7fe640 1 -- 172.21.15.29:0/1468885579 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f252c0d8b40 con 0x7f25440ad450 2024-04-25T19:44:21.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.919+0000 7f2542ffd640 1 --2- 172.21.15.29:0/1468885579 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f25240ac530 0x7f25240aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:21.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.919+0000 7f2542ffd640 1 --2- 172.21.15.29:0/1468885579 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f25240ac530 0x7f25240aea40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2534036000 tx=0x7f253403b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:21.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:21.922+0000 7f253b7fe640 1 -- 172.21.15.29:0/1468885579 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f252c021020 con 0x7f25440ad450 2024-04-25T19:44:22.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.105+0000 7f25497c6640 1 -- 172.21.15.29:0/1468885579 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f25440a7f50 con 0x7f25440ad450 2024-04-25T19:44:22.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.105+0000 7f253b7fe640 1 -- 172.21.15.29:0/1468885579 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f252c0a37b0 con 0x7f25440ad450 2024-04-25T19:44:22.106 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-04-25T19:44:22.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.107+0000 7f25497c6640 10 mgrc shutdown 2024-04-25T19:44:22.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.107+0000 7f25497c6640 1 -- 172.21.15.29:0/1468885579 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f25240ac530 msgr2=0x7f25240aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:22.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.107+0000 7f25497c6640 1 --2- 172.21.15.29:0/1468885579 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f25240ac530 0x7f25240aea40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2534036000 tx=0x7f253403b040 comp rx=0 tx=0).stop 2024-04-25T19:44:22.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.107+0000 7f25497c6640 1 -- 172.21.15.29:0/1468885579 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f25440ad450 msgr2=0x7f2544145770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:22.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.107+0000 7f25497c6640 1 --2- 172.21.15.29:0/1468885579 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f25440ad450 0x7f2544145770 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f252c00c410 tx=0x7f252c00c8d0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.107+0000 7f25497c6640 1 -- 172.21.15.29:0/1468885579 shutdown_connections 2024-04-25T19:44:22.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.107+0000 7f25497c6640 1 --2- 172.21.15.29:0/1468885579 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f25240ac530 0x7f25240aea40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.107+0000 7f25497c6640 1 --2- 172.21.15.29:0/1468885579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25441472d0 0x7f25441419b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.107+0000 7f25497c6640 1 --2- 172.21.15.29:0/1468885579 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f25440ad450 0x7f2544145770 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.107+0000 7f25497c6640 1 --2- 172.21.15.29:0/1468885579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f254406da90 0x7f25441401f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.107+0000 7f25497c6640 1 -- 172.21.15.29:0/1468885579 >> 172.21.15.29:0/1468885579 conn(0x7f25440bc3b0 msgr2=0x7f25440a63a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:22.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.107+0000 7f25497c6640 1 -- 172.21.15.29:0/1468885579 shutdown_connections 2024-04-25T19:44:22.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.107+0000 7f25497c6640 1 -- 172.21.15.29:0/1468885579 wait complete. 2024-04-25T19:44:22.120 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:44:22.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.232+0000 7f220f59e640 1 Processor -- start 2024-04-25T19:44:22.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.232+0000 7f220f59e640 1 -- start start 2024-04-25T19:44:22.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.233+0000 7f220f59e640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f221006d650 0x7f22100ace90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:22.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.233+0000 7f220f59e640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f221014f500 0x7f2210151920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:22.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.233+0000 7f220f59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22100ad880 con 0x7f22100ad3d0 2024-04-25T19:44:22.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.233+0000 7f220f59e640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f22100ae000 con 0x7f221006d650 2024-04-25T19:44:22.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.233+0000 7f220f59e640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f221014a2f0 con 0x7f221014f500 2024-04-25T19:44:22.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.234+0000 7f220ed9d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f221014f500 0x7f2210151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:22.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.234+0000 7f220dd9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f22100ad3d0 0x7f221014a7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52974/0 (socket says 172.21.15.29:52974) 2024-04-25T19:44:22.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.234+0000 7f220dd9b640 1 -- 172.21.15.29:0/1383884501 learned_addr learned my addr 172.21.15.29:0/1383884501 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:44:22.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.234+0000 7f220e59c640 1 --2- 172.21.15.29:0/1383884501 >> v2:172.21.15.102:3300/0 conn(0x7f221006d650 0x7f22100ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:22.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.234+0000 7f220d59a640 1 -- 172.21.15.29:0/1383884501 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2471317519 0 0) 0x7f21f8001110 con 0x7f22100ad3d0 2024-04-25T19:44:22.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.234+0000 7f220d59a640 1 -- 172.21.15.29:0/1383884501 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21fc003520 con 0x7f22100ad3d0 2024-04-25T19:44:22.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.234+0000 7f220ed9d640 1 -- 172.21.15.29:0/1383884501 >> v2:172.21.15.102:3300/0 conn(0x7f221006d650 msgr2=0x7f22100ace90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:22.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.234+0000 7f220ed9d640 1 --2- 172.21.15.29:0/1383884501 >> v2:172.21.15.102:3300/0 conn(0x7f221006d650 0x7f22100ace90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.234+0000 7f220ed9d640 1 -- 172.21.15.29:0/1383884501 >> v1:172.21.15.29:6789/0 conn(0x7f22100ad3d0 legacy=0x7f221014a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:22.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.234+0000 7f220ed9d640 1 -- 172.21.15.29:0/1383884501 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22100ae490 con 0x7f221014f500 2024-04-25T19:44:22.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.235+0000 7f220ed9d640 1 --2- 172.21.15.29:0/1383884501 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f221014f500 0x7f2210151920 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2200009d90 tx=0x7f220002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd1fc635eda0c363 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:22.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.235+0000 7f220d59a640 1 -- 172.21.15.29:0/1383884501 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2200039070 con 0x7f221014f500 2024-04-25T19:44:22.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.235+0000 7f220d59a640 1 -- 172.21.15.29:0/1383884501 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f220002cc50 con 0x7f221014f500 2024-04-25T19:44:22.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.235+0000 7f220d59a640 1 -- 172.21.15.29:0/1383884501 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2200034440 con 0x7f221014f500 2024-04-25T19:44:22.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.235+0000 7f220f59e640 1 -- 172.21.15.29:0/1383884501 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f221014f500 msgr2=0x7f2210151920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:22.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.235+0000 7f220f59e640 1 --2- 172.21.15.29:0/1383884501 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f221014f500 0x7f2210151920 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2200009d90 tx=0x7f220002bea0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.236+0000 7f220f59e640 1 -- 172.21.15.29:0/1383884501 shutdown_connections 2024-04-25T19:44:22.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.236+0000 7f220f59e640 1 --2- 172.21.15.29:0/1383884501 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f221014f500 0x7f2210151920 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.236+0000 7f220f59e640 1 --2- 172.21.15.29:0/1383884501 >> v2:172.21.15.102:3300/0 conn(0x7f221006d650 0x7f22100ace90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.236+0000 7f220f59e640 1 -- 172.21.15.29:0/1383884501 >> 172.21.15.29:0/1383884501 conn(0x7f221006cb30 msgr2=0x7f221006cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:22.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.236+0000 7f220f59e640 1 -- 172.21.15.29:0/1383884501 shutdown_connections 2024-04-25T19:44:22.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.236+0000 7f220f59e640 1 -- 172.21.15.29:0/1383884501 wait complete. 2024-04-25T19:44:22.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.237+0000 7f220f59e640 1 Processor -- start 2024-04-25T19:44:22.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.237+0000 7f220f59e640 1 -- start start 2024-04-25T19:44:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.238+0000 7f220f59e640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f221006d650 0x7f22101e0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.238+0000 7f220f59e640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f22100ad3d0 0x7f221013a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.238+0000 7f220f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f221014f500 0x7f22101dd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.238+0000 7f220f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f221006c930 con 0x7f221014f500 2024-04-25T19:44:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.238+0000 7f220f59e640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f221006c7b0 con 0x7f221006d650 2024-04-25T19:44:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.238+0000 7f220f59e640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f221006da60 con 0x7f22100ad3d0 2024-04-25T19:44:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.238+0000 7f220dd9b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f22100ad3d0 0x7f221013a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.239+0000 7f220ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f221014f500 0x7f22101dd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.239+0000 7f220dd9b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f22100ad3d0 0x7f221013a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:35296/0 (socket says 172.21.15.29:35296) 2024-04-25T19:44:22.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.239+0000 7f220dd9b640 1 -- 172.21.15.29:0/1080640531 learned_addr learned my addr 172.21.15.29:0/1080640531 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:22.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.239+0000 7f220e59c640 1 --2- 172.21.15.29:0/1080640531 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f221006d650 0x7f22101e0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:22.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.239+0000 7f220ed9d640 1 -- 172.21.15.29:0/1080640531 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f22100ad3d0 msgr2=0x7f221013a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:22.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.239+0000 7f220ed9d640 1 --2- 172.21.15.29:0/1080640531 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f22100ad3d0 0x7f221013a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.239+0000 7f220ed9d640 1 -- 172.21.15.29:0/1080640531 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f221006d650 msgr2=0x7f22101e0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:22.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.239+0000 7f220ed9d640 1 --2- 172.21.15.29:0/1080640531 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f221006d650 0x7f22101e0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.239+0000 7f220ed9d640 1 -- 172.21.15.29:0/1080640531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f221013ac30 con 0x7f221014f500 2024-04-25T19:44:22.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.239+0000 7f220dd9b640 1 --2- 172.21.15.29:0/1080640531 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f22100ad3d0 0x7f221013a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:22.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.239+0000 7f220ed9d640 1 --2- 172.21.15.29:0/1080640531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f221014f500 0x7f22101dd430 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2200033000 tx=0x7f2200004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:22.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.239+0000 7f2206ffd640 1 -- 172.21.15.29:0/1080640531 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2200039050 con 0x7f221014f500 2024-04-25T19:44:22.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.239+0000 7f2206ffd640 1 -- 172.21.15.29:0/1080640531 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2200045070 con 0x7f221014f500 2024-04-25T19:44:22.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.239+0000 7f220f59e640 1 -- 172.21.15.29:0/1080640531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f221013af30 con 0x7f221014f500 2024-04-25T19:44:22.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.239+0000 7f2206ffd640 1 -- 172.21.15.29:0/1080640531 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2200009450 con 0x7f221014f500 2024-04-25T19:44:22.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.239+0000 7f220f59e640 1 -- 172.21.15.29:0/1080640531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22101ddb50 con 0x7f221014f500 2024-04-25T19:44:22.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.240+0000 7f220f59e640 1 -- 172.21.15.29:0/1080640531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22101de1e0 con 0x7f221014f500 2024-04-25T19:44:22.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.242+0000 7f2206ffd640 1 -- 172.21.15.29:0/1080640531 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f220003e610 con 0x7f221014f500 2024-04-25T19:44:22.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.242+0000 7f2206ffd640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:22.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.242+0000 7f2206ffd640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:22.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.242+0000 7f2206ffd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:22.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.242+0000 7f2206ffd640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:22.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.242+0000 7f2206ffd640 1 --2- 172.21.15.29:0/1080640531 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f21f00ac560 0x7f21f00aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:22.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.242+0000 7f2206ffd640 1 -- 172.21.15.29:0/1080640531 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f22000fbd30 con 0x7f221014f500 2024-04-25T19:44:22.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.243+0000 7f220e59c640 1 --2- 172.21.15.29:0/1080640531 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f21f00ac560 0x7f21f00aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:22.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.243+0000 7f220e59c640 1 --2- 172.21.15.29:0/1080640531 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f21f00ac560 0x7f21f00aea70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2208006fd0 tx=0x7f2208008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:22.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.247+0000 7f2206ffd640 1 -- 172.21.15.29:0/1080640531 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f22000c69a0 con 0x7f221014f500 2024-04-25T19:44:22.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.429+0000 7f220f59e640 1 -- 172.21.15.29:0/1080640531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f22101e1480 con 0x7f221014f500 2024-04-25T19:44:22.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.429+0000 7f2206ffd640 1 -- 172.21.15.29:0/1080640531 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f22000c62a0 con 0x7f221014f500 2024-04-25T19:44:22.431 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-04-25T19:44:22.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.431+0000 7f220f59e640 10 mgrc shutdown 2024-04-25T19:44:22.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.431+0000 7f220f59e640 1 -- 172.21.15.29:0/1080640531 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f21f00ac560 msgr2=0x7f21f00aea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:22.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.431+0000 7f220f59e640 1 --2- 172.21.15.29:0/1080640531 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f21f00ac560 0x7f21f00aea70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2208006fd0 tx=0x7f2208008040 comp rx=0 tx=0).stop 2024-04-25T19:44:22.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.431+0000 7f220f59e640 1 -- 172.21.15.29:0/1080640531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f221014f500 msgr2=0x7f22101dd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:22.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.431+0000 7f220f59e640 1 --2- 172.21.15.29:0/1080640531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f221014f500 0x7f22101dd430 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2200033000 tx=0x7f2200004a40 comp rx=0 tx=0).stop 2024-04-25T19:44:22.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.431+0000 7f220f59e640 1 -- 172.21.15.29:0/1080640531 shutdown_connections 2024-04-25T19:44:22.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.431+0000 7f220f59e640 1 --2- 172.21.15.29:0/1080640531 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f21f00ac560 0x7f21f00aea70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.431+0000 7f220f59e640 1 --2- 172.21.15.29:0/1080640531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f221014f500 0x7f22101dd430 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.431+0000 7f220f59e640 1 --2- 172.21.15.29:0/1080640531 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f22100ad3d0 0x7f221013a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.431+0000 7f220f59e640 1 --2- 172.21.15.29:0/1080640531 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f221006d650 0x7f22101e0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.431+0000 7f220f59e640 1 -- 172.21.15.29:0/1080640531 >> 172.21.15.29:0/1080640531 conn(0x7f221006cb30 msgr2=0x7f2210143540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:22.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.431+0000 7f220f59e640 1 -- 172.21.15.29:0/1080640531 shutdown_connections 2024-04-25T19:44:22.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.431+0000 7f220f59e640 1 -- 172.21.15.29:0/1080640531 wait complete. 2024-04-25T19:44:22.444 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:44:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.553+0000 7f5adfb7c640 1 Processor -- start 2024-04-25T19:44:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.553+0000 7f5adfb7c640 1 -- start start 2024-04-25T19:44:22.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.554+0000 7f5adfb7c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5ad8073b90 0x7f5ad80a8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:22.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.554+0000 7f5adfb7c640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5ad806ff10 0x7f5ad8070320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:22.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.554+0000 7f5adfb7c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5ad8147fc0 0x7f5ad81483d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:22.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.554+0000 7f5adfb7c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f5ad8073fa0 con 0x7f5ad8147fc0 2024-04-25T19:44:22.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.554+0000 7f5adfb7c640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f5ad8070ba0 con 0x7f5ad806ff10 2024-04-25T19:44:22.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.554+0000 7f5adfb7c640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f5ad8144940 con 0x7f5ad8073b90 2024-04-25T19:44:22.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.554+0000 7f5add918640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5ad8073b90 0x7f5ad80a8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:22.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.554+0000 7f5ade119640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5ad8147fc0 0x7f5ad81483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.554+0000 7f5add918640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5ad8073b90 0x7f5ad80a8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:35308/0 (socket says 172.21.15.29:35308) 2024-04-25T19:44:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.554+0000 7f5ade119640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5ad8147fc0 0x7f5ad81483d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39912/0 (socket says 172.21.15.29:39912) 2024-04-25T19:44:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.554+0000 7f5add918640 1 -- 172.21.15.29:0/704350103 learned_addr learned my addr 172.21.15.29:0/704350103 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.555+0000 7f5add117640 1 --2- 172.21.15.29:0/704350103 >> v2:172.21.15.102:3300/0 conn(0x7f5ad806ff10 0x7f5ad8070320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.555+0000 7f5add918640 1 -- 172.21.15.29:0/704350103 >> v2:172.21.15.102:3300/0 conn(0x7f5ad806ff10 msgr2=0x7f5ad8070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.555+0000 7f5add918640 1 --2- 172.21.15.29:0/704350103 >> v2:172.21.15.102:3300/0 conn(0x7f5ad806ff10 0x7f5ad8070320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.555+0000 7f5add918640 1 -- 172.21.15.29:0/704350103 >> v2:172.21.15.29:3300/0 conn(0x7f5ad8147fc0 msgr2=0x7f5ad81483d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.555+0000 7f5add918640 1 --2- 172.21.15.29:0/704350103 >> v2:172.21.15.29:3300/0 conn(0x7f5ad8147fc0 0x7f5ad81483d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.555+0000 7f5add918640 1 -- 172.21.15.29:0/704350103 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad8070860 con 0x7f5ad8073b90 2024-04-25T19:44:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.555+0000 7f5ade119640 1 --2- 172.21.15.29:0/704350103 >> v2:172.21.15.29:3300/0 conn(0x7f5ad8147fc0 0x7f5ad81483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.555+0000 7f5add918640 1 --2- 172.21.15.29:0/704350103 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5ad8073b90 0x7f5ad80a8490 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5ad80a9f90 tx=0x7f5ad402b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3d3b57c5114f7bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.556+0000 7f5adc916640 1 -- 172.21.15.29:0/704350103 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad40317d0 con 0x7f5ad8073b90 2024-04-25T19:44:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.556+0000 7f5adc916640 1 -- 172.21.15.29:0/704350103 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ad4037070 con 0x7f5ad8073b90 2024-04-25T19:44:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.556+0000 7f5adc916640 1 -- 172.21.15.29:0/704350103 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad4031b90 con 0x7f5ad8073b90 2024-04-25T19:44:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.556+0000 7f5adfb7c640 1 -- 172.21.15.29:0/704350103 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5ad8073b90 msgr2=0x7f5ad80a8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.556+0000 7f5adfb7c640 1 --2- 172.21.15.29:0/704350103 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5ad8073b90 0x7f5ad80a8490 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5ad80a9f90 tx=0x7f5ad402b870 comp rx=0 tx=0).stop 2024-04-25T19:44:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.556+0000 7f5adfb7c640 1 -- 172.21.15.29:0/704350103 shutdown_connections 2024-04-25T19:44:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.556+0000 7f5adfb7c640 1 --2- 172.21.15.29:0/704350103 >> v2:172.21.15.102:3300/0 conn(0x7f5ad806ff10 0x7f5ad8070320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.556+0000 7f5adfb7c640 1 --2- 172.21.15.29:0/704350103 >> v2:172.21.15.29:3300/0 conn(0x7f5ad8147fc0 0x7f5ad81483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.556+0000 7f5adfb7c640 1 --2- 172.21.15.29:0/704350103 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5ad8073b90 0x7f5ad80a8490 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.556+0000 7f5adfb7c640 1 -- 172.21.15.29:0/704350103 >> 172.21.15.29:0/704350103 conn(0x7f5ad80a9390 msgr2=0x7f5ad80a97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:22.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.557+0000 7f5adfb7c640 1 -- 172.21.15.29:0/704350103 shutdown_connections 2024-04-25T19:44:22.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.557+0000 7f5adfb7c640 1 -- 172.21.15.29:0/704350103 wait complete. 2024-04-25T19:44:22.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.558+0000 7f5adfb7c640 1 Processor -- start 2024-04-25T19:44:22.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.558+0000 7f5adfb7c640 1 -- start start 2024-04-25T19:44:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.559+0000 7f5adfb7c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ad806ff10 0x7f5ad81d1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.559+0000 7f5adfb7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ad8073b90 0x7f5ad8141ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.559+0000 7f5adfb7c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5ad8147fc0 0x7f5ad81dcd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.559+0000 7f5adfb7c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad80ab8c0 con 0x7f5ad8073b90 2024-04-25T19:44:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.559+0000 7f5adfb7c640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad80ab740 con 0x7f5ad806ff10 2024-04-25T19:44:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.559+0000 7f5adfb7c640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f5ad80aac70 con 0x7f5ad8147fc0 2024-04-25T19:44:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.559+0000 7f5add117640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ad8073b90 0x7f5ad8141ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.559+0000 7f5ade119640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5ad8147fc0 0x7f5ad81dcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.559+0000 7f5add117640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ad8073b90 0x7f5ad8141ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39918/0 (socket says 172.21.15.29:39918) 2024-04-25T19:44:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.559+0000 7f5add117640 1 -- 172.21.15.29:0/4159558421 learned_addr learned my addr 172.21.15.29:0/4159558421 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.559+0000 7f5add918640 1 --2- 172.21.15.29:0/4159558421 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ad806ff10 0x7f5ad81d1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.559+0000 7f5add117640 1 -- 172.21.15.29:0/4159558421 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5ad8147fc0 msgr2=0x7f5ad81dcd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.559+0000 7f5add117640 1 --2- 172.21.15.29:0/4159558421 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5ad8147fc0 0x7f5ad81dcd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.559+0000 7f5add117640 1 -- 172.21.15.29:0/4159558421 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ad806ff10 msgr2=0x7f5ad81d1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.559+0000 7f5add117640 1 --2- 172.21.15.29:0/4159558421 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ad806ff10 0x7f5ad81d1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.559+0000 7f5add117640 1 -- 172.21.15.29:0/4159558421 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad8141910 con 0x7f5ad8073b90 2024-04-25T19:44:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.559+0000 7f5ade119640 1 --2- 172.21.15.29:0/4159558421 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5ad8147fc0 0x7f5ad81dcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.559+0000 7f5add918640 1 --2- 172.21.15.29:0/4159558421 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ad806ff10 0x7f5ad81d1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:44:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.560+0000 7f5add117640 1 --2- 172.21.15.29:0/4159558421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ad8073b90 0x7f5ad8141ef0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800e870 tx=0x7f5ac800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.560+0000 7f5ace7fc640 1 -- 172.21.15.29:0/4159558421 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8017070 con 0x7f5ad8073b90 2024-04-25T19:44:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.560+0000 7f5ace7fc640 1 -- 172.21.15.29:0/4159558421 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ac800ee60 con 0x7f5ad8073b90 2024-04-25T19:44:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.560+0000 7f5adfb7c640 1 -- 172.21.15.29:0/4159558421 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ad8143430 con 0x7f5ad8073b90 2024-04-25T19:44:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.560+0000 7f5ace7fc640 1 -- 172.21.15.29:0/4159558421 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac800cb60 con 0x7f5ad8073b90 2024-04-25T19:44:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.560+0000 7f5adfb7c640 1 -- 172.21.15.29:0/4159558421 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ad81d1c60 con 0x7f5ad8073b90 2024-04-25T19:44:22.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.561+0000 7f5adfb7c640 1 -- 172.21.15.29:0/4159558421 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ad80507b0 con 0x7f5ad8073b90 2024-04-25T19:44:22.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.562+0000 7f5ace7fc640 1 -- 172.21.15.29:0/4159558421 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f5ac801a070 con 0x7f5ad8073b90 2024-04-25T19:44:22.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.562+0000 7f5ace7fc640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:22.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.563+0000 7f5ace7fc640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:22.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.563+0000 7f5ace7fc640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:22.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.563+0000 7f5ace7fc640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:22.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.563+0000 7f5ace7fc640 1 --2- 172.21.15.29:0/4159558421 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5abc0ac530 0x7f5abc0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:22.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.563+0000 7f5ace7fc640 1 -- 172.21.15.29:0/4159558421 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f5ac80d9950 con 0x7f5ad8073b90 2024-04-25T19:44:22.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.563+0000 7f5add918640 1 --2- 172.21.15.29:0/4159558421 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5abc0ac530 0x7f5abc0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:22.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.564+0000 7f5add918640 1 --2- 172.21.15.29:0/4159558421 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5abc0ac530 0x7f5abc0aea40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4036000 tx=0x7f5ad4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:22.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.566+0000 7f5ace7fc640 1 -- 172.21.15.29:0/4159558421 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5ac80a45c0 con 0x7f5ad8073b90 2024-04-25T19:44:22.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.791+0000 7f5adfb7c640 1 -- 172.21.15.29:0/4159558421 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ad8071970 con 0x7f5ad8073b90 2024-04-25T19:44:22.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.792+0000 7f5ace7fc640 1 -- 172.21.15.29:0/4159558421 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+4351 (secure 0 0 0) 0x7f5ac80a3ec0 con 0x7f5ad8073b90 2024-04-25T19:44:22.793 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:22.793 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6809/3582557422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3582557422},{"type":"v1","addr":"172.21.15.29:6809","nonce":3582557422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4234,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/1228868047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1228868047},{"type":"v1","addr":"172.21.15.102:6827","nonce":1228868047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:44:09.410562+0000","modified":"2024-04-25T19:44:12.425418+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6811/844243878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":844243878},{"type":"v1","addr":"172.21.15.29:6811","nonce":844243878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":1}]} 2024-04-25T19:44:22.793 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2024-04-25T19:44:22.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.794+0000 7f5adfb7c640 10 mgrc shutdown 2024-04-25T19:44:22.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.794+0000 7f5adfb7c640 1 -- 172.21.15.29:0/4159558421 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5abc0ac530 msgr2=0x7f5abc0aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.794+0000 7f5adfb7c640 1 --2- 172.21.15.29:0/4159558421 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5abc0ac530 0x7f5abc0aea40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4036000 tx=0x7f5ad4008030 comp rx=0 tx=0).stop 2024-04-25T19:44:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.794+0000 7f5adfb7c640 1 -- 172.21.15.29:0/4159558421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ad8073b90 msgr2=0x7f5ad8141ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.794+0000 7f5adfb7c640 1 --2- 172.21.15.29:0/4159558421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ad8073b90 0x7f5ad8141ef0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800e870 tx=0x7f5ac800ec70 comp rx=0 tx=0).stop 2024-04-25T19:44:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.794+0000 7f5adfb7c640 1 -- 172.21.15.29:0/4159558421 shutdown_connections 2024-04-25T19:44:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.794+0000 7f5adfb7c640 1 --2- 172.21.15.29:0/4159558421 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f5abc0ac530 0x7f5abc0aea40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.794+0000 7f5adfb7c640 1 --2- 172.21.15.29:0/4159558421 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5ad8147fc0 0x7f5ad81dcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.794+0000 7f5adfb7c640 1 --2- 172.21.15.29:0/4159558421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ad8073b90 0x7f5ad8141ef0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.794+0000 7f5adfb7c640 1 --2- 172.21.15.29:0/4159558421 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ad806ff10 0x7f5ad81d1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.794+0000 7f5adfb7c640 1 -- 172.21.15.29:0/4159558421 >> 172.21.15.29:0/4159558421 conn(0x7f5ad80a9390 msgr2=0x7f5ad80b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.794+0000 7f5adfb7c640 1 -- 172.21.15.29:0/4159558421 shutdown_connections 2024-04-25T19:44:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.794+0000 7f5adfb7c640 1 -- 172.21.15.29:0/4159558421 wait complete. 2024-04-25T19:44:22.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:44:22.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.916+0000 7fea221f8640 1 Processor -- start 2024-04-25T19:44:22.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.916+0000 7fea221f8640 1 -- start start 2024-04-25T19:44:22.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.917+0000 7fea221f8640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fea1c144240 0x7fea1c146c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:22.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.917+0000 7fea221f8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fea1c147150 0x7fea1c151590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:22.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.917+0000 7fea221f8640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea1c144650 con 0x7fea1c143e30 2024-04-25T19:44:22.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.917+0000 7fea221f8640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fea1c147560 con 0x7fea1c147150 2024-04-25T19:44:22.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.917+0000 7fea221f8640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fea1c0a5a90 con 0x7fea1c144240 2024-04-25T19:44:22.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.917+0000 7fea1bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fea1c147150 0x7fea1c151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:22.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.917+0000 7fea1bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fea1c147150 0x7fea1c151590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39926/0 (socket says 172.21.15.29:39926) 2024-04-25T19:44:22.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.917+0000 7fea1bfff640 1 -- 172.21.15.29:0/2096738796 learned_addr learned my addr 172.21.15.29:0/2096738796 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:22.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.917+0000 7fea1affd640 1 --2- 172.21.15.29:0/2096738796 >> v2:172.21.15.102:3300/0 conn(0x7fea1c144240 0x7fea1c146c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:22.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.917+0000 7fea1bfff640 1 -- 172.21.15.29:0/2096738796 >> v2:172.21.15.102:3300/0 conn(0x7fea1c144240 msgr2=0x7fea1c146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:22.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.917+0000 7fea1bfff640 1 --2- 172.21.15.29:0/2096738796 >> v2:172.21.15.102:3300/0 conn(0x7fea1c144240 0x7fea1c146c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.917+0000 7fea1bfff640 1 -- 172.21.15.29:0/2096738796 >> v1:172.21.15.102:6789/0 conn(0x7fea1c143e30 legacy=0x7fea1c0acaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:22.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.917+0000 7fea1bfff640 1 -- 172.21.15.29:0/2096738796 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea1c147790 con 0x7fea1c147150 2024-04-25T19:44:22.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.918+0000 7fea1bfff640 1 --2- 172.21.15.29:0/2096738796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea1c147150 0x7fea1c151590 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fea1c0adbc0 tx=0x7fea1402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb0b909a7c59a936 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:22.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.918+0000 7fea1a7fc640 1 -- 172.21.15.29:0/2096738796 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1402ca80 con 0x7fea1c147150 2024-04-25T19:44:22.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.918+0000 7fea1a7fc640 1 -- 172.21.15.29:0/2096738796 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea1402cc20 con 0x7fea1c147150 2024-04-25T19:44:22.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.918+0000 7fea1a7fc640 1 -- 172.21.15.29:0/2096738796 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea140343f0 con 0x7fea1c147150 2024-04-25T19:44:22.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.919+0000 7fea221f8640 1 -- 172.21.15.29:0/2096738796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea1c147150 msgr2=0x7fea1c151590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:22.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.919+0000 7fea221f8640 1 --2- 172.21.15.29:0/2096738796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea1c147150 0x7fea1c151590 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fea1c0adbc0 tx=0x7fea1402bee0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.919+0000 7fea221f8640 1 -- 172.21.15.29:0/2096738796 shutdown_connections 2024-04-25T19:44:22.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.919+0000 7fea221f8640 1 --2- 172.21.15.29:0/2096738796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea1c147150 0x7fea1c151590 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.919+0000 7fea221f8640 1 --2- 172.21.15.29:0/2096738796 >> v2:172.21.15.102:3300/0 conn(0x7fea1c144240 0x7fea1c146c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.919+0000 7fea221f8640 1 -- 172.21.15.29:0/2096738796 >> 172.21.15.29:0/2096738796 conn(0x7fea1c0aafc0 msgr2=0x7fea1c0ab3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:22.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.919+0000 7fea221f8640 1 -- 172.21.15.29:0/2096738796 shutdown_connections 2024-04-25T19:44:22.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.919+0000 7fea221f8640 1 -- 172.21.15.29:0/2096738796 wait complete. 2024-04-25T19:44:22.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.920+0000 7fea221f8640 1 Processor -- start 2024-04-25T19:44:22.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.921+0000 7fea221f8640 1 -- start start 2024-04-25T19:44:22.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.921+0000 7fea221f8640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fea1c143e30 0x7fea1c149fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:22.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea221f8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea1c144240 0x7fea1c14fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:22.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea221f8640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea1c147150 0x7fea1c14b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:22.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea221f8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fea1c0ad330 con 0x7fea1c144240 2024-04-25T19:44:22.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea221f8640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fea1c0ad1b0 con 0x7fea1c147150 2024-04-25T19:44:22.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea221f8640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fea1c151ad0 con 0x7fea1c143e30 2024-04-25T19:44:22.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea1affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea1c144240 0x7fea1c14fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:22.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea1b7fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fea1c143e30 0x7fea1c149fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:22.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea1affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea1c144240 0x7fea1c14fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39938/0 (socket says 172.21.15.29:39938) 2024-04-25T19:44:22.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea1affd640 1 -- 172.21.15.29:0/690493598 learned_addr learned my addr 172.21.15.29:0/690493598 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:22.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea1bfff640 1 --2- 172.21.15.29:0/690493598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea1c147150 0x7fea1c14b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:22.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea1affd640 1 -- 172.21.15.29:0/690493598 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fea1c143e30 msgr2=0x7fea1c149fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:22.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea1affd640 1 --2- 172.21.15.29:0/690493598 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fea1c143e30 0x7fea1c149fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea1affd640 1 -- 172.21.15.29:0/690493598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea1c147150 msgr2=0x7fea1c14b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:22.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea1affd640 1 --2- 172.21.15.29:0/690493598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea1c147150 0x7fea1c14b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:22.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea1affd640 1 -- 172.21.15.29:0/690493598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea1c14f730 con 0x7fea1c144240 2024-04-25T19:44:22.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea1b7fe640 1 --2- 172.21.15.29:0/690493598 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fea1c143e30 0x7fea1c149fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:22.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea1affd640 1 --2- 172.21.15.29:0/690493598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea1c144240 0x7fea1c14fbf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fea1000e8a0 tx=0x7fea1000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:22.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea03fff640 1 -- 172.21.15.29:0/690493598 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea10017070 con 0x7fea1c144240 2024-04-25T19:44:22.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea03fff640 1 -- 172.21.15.29:0/690493598 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea1000c8e0 con 0x7fea1c144240 2024-04-25T19:44:22.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea221f8640 1 -- 172.21.15.29:0/690493598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea1c14aa50 con 0x7fea1c144240 2024-04-25T19:44:22.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea03fff640 1 -- 172.21.15.29:0/690493598 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1000ccd0 con 0x7fea1c144240 2024-04-25T19:44:22.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.922+0000 7fea221f8640 1 -- 172.21.15.29:0/690493598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea1c13e240 con 0x7fea1c144240 2024-04-25T19:44:22.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.923+0000 7fea221f8640 1 -- 172.21.15.29:0/690493598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea1c0507b0 con 0x7fea1c144240 2024-04-25T19:44:22.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.925+0000 7fea03fff640 1 -- 172.21.15.29:0/690493598 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7fea1001a070 con 0x7fea1c144240 2024-04-25T19:44:22.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.925+0000 7fea03fff640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:22.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.925+0000 7fea03fff640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:22.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.925+0000 7fea03fff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:22.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.925+0000 7fea03fff640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:22.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.925+0000 7fea03fff640 1 --2- 172.21.15.29:0/690493598 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fea080ac530 0x7fea080aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:22.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.925+0000 7fea03fff640 1 -- 172.21.15.29:0/690493598 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7fea100d98e0 con 0x7fea1c144240 2024-04-25T19:44:22.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.925+0000 7fea1b7fe640 1 --2- 172.21.15.29:0/690493598 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fea080ac530 0x7fea080aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:22.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.926+0000 7fea1b7fe640 1 --2- 172.21.15.29:0/690493598 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fea080ac530 0x7fea080aea40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fea0c001ad0 tx=0x7fea0c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:22.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:22.929+0000 7fea03fff640 1 -- 172.21.15.29:0/690493598 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fea100a3f00 con 0x7fea1c144240 2024-04-25T19:44:23.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.118+0000 7fea221f8640 1 -- 172.21.15.29:0/690493598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fea1c146f30 con 0x7fea1c144240 2024-04-25T19:44:23.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.119+0000 7fea03fff640 1 -- 172.21.15.29:0/690493598 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+13512 (secure 0 0 0) 0x7fea100a8200 con 0x7fea1c144240 2024-04-25T19:44:23.120 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:23.120 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"1b3c39b5-e787-409e-b0bd-f4f491c89d72","created":"2024-04-25T19:43:47.854697+0000","modified":"2024-04-25T19:44:11.035019+0000","last_up_change":"2024-04-25T19:43:57.853237+0000","last_in_change":"2024-04-25T19:43:49.022776+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:43:58.096700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T19:44:03.617011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:44:07.890493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2024-04-25T19:44:08.355570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d805223-ee84-494f-bd31-59091906aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6801","nonce":695704147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6803","nonce":695704147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6807","nonce":695704147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6805","nonce":695704147}]},"public_addr":"172.21.15.29:6801/695704147","cluster_addr":"172.21.15.29:6803/695704147","heartbeat_back_addr":"172.21.15.29:6807/695704147","heartbeat_front_addr":"172.21.15.29:6805/695704147","state":["exists","up"]},{"osd":1,"uuid":"607c5038-38af-440f-8371-e71758549920","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6814","nonce":3135821452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6817","nonce":3135821452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6823","nonce":3135821452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6821","nonce":3135821452}]},"public_addr":"172.21.15.102:6814/3135821452","cluster_addr":"172.21.15.102:6817/3135821452","heartbeat_back_addr":"172.21.15.102:6823/3135821452","heartbeat_front_addr":"172.21.15.102:6821/3135821452","state":["exists","up"]},{"osd":2,"uuid":"6859ae76-54bb-4c55-b0dc-5e2735c0d31c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6801","nonce":2071844144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6803","nonce":2071844144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6811","nonce":2071844144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6806","nonce":2071844144}]},"public_addr":"172.21.15.102:6801/2071844144","cluster_addr":"172.21.15.102:6803/2071844144","heartbeat_back_addr":"172.21.15.102:6811/2071844144","heartbeat_front_addr":"172.21.15.102:6806/2071844144","state":["exists","up"]},{"osd":3,"uuid":"a98b7b88-5206-4f22-84a6-78212e0d5f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6805","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6807","nonce":339944984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6810","nonce":339944984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6820","nonce":339944984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6815","nonce":339944984}]},"public_addr":"172.21.15.102:6807/339944984","cluster_addr":"172.21.15.102:6810/339944984","heartbeat_back_addr":"172.21.15.102:6820/339944984","heartbeat_front_addr":"172.21.15.102:6815/339944984","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.121529+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.201155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.264387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.198713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:44:23.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.121+0000 7fea221f8640 10 mgrc shutdown 2024-04-25T19:44:23.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.121+0000 7fea221f8640 1 -- 172.21.15.29:0/690493598 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fea080ac530 msgr2=0x7fea080aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:23.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.121+0000 7fea221f8640 1 --2- 172.21.15.29:0/690493598 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fea080ac530 0x7fea080aea40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fea0c001ad0 tx=0x7fea0c009040 comp rx=0 tx=0).stop 2024-04-25T19:44:23.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.121+0000 7fea221f8640 1 -- 172.21.15.29:0/690493598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea1c144240 msgr2=0x7fea1c14fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:23.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.121+0000 7fea221f8640 1 --2- 172.21.15.29:0/690493598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea1c144240 0x7fea1c14fbf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fea1000e8a0 tx=0x7fea1000ed60 comp rx=0 tx=0).stop 2024-04-25T19:44:23.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.121+0000 7fea221f8640 1 -- 172.21.15.29:0/690493598 shutdown_connections 2024-04-25T19:44:23.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.121+0000 7fea221f8640 1 --2- 172.21.15.29:0/690493598 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7fea080ac530 0x7fea080aea40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.121+0000 7fea221f8640 1 --2- 172.21.15.29:0/690493598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea1c147150 0x7fea1c14b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.121+0000 7fea221f8640 1 --2- 172.21.15.29:0/690493598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea1c144240 0x7fea1c14fbf0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.121+0000 7fea221f8640 1 --2- 172.21.15.29:0/690493598 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fea1c143e30 0x7fea1c149fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.121+0000 7fea221f8640 1 -- 172.21.15.29:0/690493598 >> 172.21.15.29:0/690493598 conn(0x7fea1c0aafc0 msgr2=0x7fea1c143220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:23.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.121+0000 7fea221f8640 1 -- 172.21.15.29:0/690493598 shutdown_connections 2024-04-25T19:44:23.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.121+0000 7fea221f8640 1 -- 172.21.15.29:0/690493598 wait complete. 2024-04-25T19:44:23.134 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:44:23.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.243+0000 7ff4ce7cf640 1 Processor -- start 2024-04-25T19:44:23.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.243+0000 7ff4ce7cf640 1 -- start start 2024-04-25T19:44:23.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.244+0000 7ff4ce7cf640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff4c8143e30 0x7ff4c8144240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.244+0000 7ff4ce7cf640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff4c80acaa0 0x7ff4c80aceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.244+0000 7ff4ce7cf640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff4c8146c10 0x7ff4c8147020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.244+0000 7ff4ce7cf640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7ff4c8147560 con 0x7ff4c80acaa0 2024-04-25T19:44:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.244+0000 7ff4ce7cf640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff4c80a5a90 con 0x7ff4c8146c10 2024-04-25T19:44:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.244+0000 7ff4ce7cf640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c8147790 con 0x7ff4c8143e30 2024-04-25T19:44:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.244+0000 7ff4c7fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff4c8143e30 0x7ff4c8144240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.244+0000 7ff4c77fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff4c80acaa0 0x7ff4c80aceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.244+0000 7ff4c7fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff4c8143e30 0x7ff4c8144240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:35336/0 (socket says 172.21.15.29:35336) 2024-04-25T19:44:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.244+0000 7ff4c7fff640 1 -- 172.21.15.29:0/2037978752 learned_addr learned my addr 172.21.15.29:0/2037978752 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.244+0000 7ff4ccd6c640 1 --2- 172.21.15.29:0/2037978752 >> v2:172.21.15.102:3300/0 conn(0x7ff4c8146c10 0x7ff4c8147020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.244+0000 7ff4c77fe640 1 -- 172.21.15.29:0/2037978752 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff4c8143e30 msgr2=0x7ff4c8144240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.244+0000 7ff4c77fe640 1 --2- 172.21.15.29:0/2037978752 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff4c8143e30 0x7ff4c8144240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.244+0000 7ff4c77fe640 1 -- 172.21.15.29:0/2037978752 >> v2:172.21.15.102:3300/0 conn(0x7ff4c8146c10 msgr2=0x7ff4c8147020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.244+0000 7ff4c77fe640 1 --2- 172.21.15.29:0/2037978752 >> v2:172.21.15.102:3300/0 conn(0x7ff4c8146c10 0x7ff4c8147020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.244+0000 7ff4c77fe640 1 -- 172.21.15.29:0/2037978752 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c8147a50 con 0x7ff4c80acaa0 2024-04-25T19:44:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.244+0000 7ff4c7fff640 1 --2- 172.21.15.29:0/2037978752 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff4c8143e30 0x7ff4c8144240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:23.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.244+0000 7ff4ccd6c640 1 --2- 172.21.15.29:0/2037978752 >> v2:172.21.15.102:3300/0 conn(0x7ff4c8146c10 0x7ff4c8147020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:44:23.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.245+0000 7ff4c77fe640 1 --2- 172.21.15.29:0/2037978752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4c80acaa0 0x7ff4c80aceb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80097a0 tx=0x7ff4b802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d61433bdb8549b01 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:23.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.245+0000 7ff4c6ffd640 1 -- 172.21.15.29:0/2037978752 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8031780 con 0x7ff4c80acaa0 2024-04-25T19:44:23.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.245+0000 7ff4c6ffd640 1 -- 172.21.15.29:0/2037978752 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b8037070 con 0x7ff4c80acaa0 2024-04-25T19:44:23.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.245+0000 7ff4c6ffd640 1 -- 172.21.15.29:0/2037978752 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8031b40 con 0x7ff4c80acaa0 2024-04-25T19:44:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.246+0000 7ff4ce7cf640 1 -- 172.21.15.29:0/2037978752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4c80acaa0 msgr2=0x7ff4c80aceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.246+0000 7ff4ce7cf640 1 --2- 172.21.15.29:0/2037978752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4c80acaa0 0x7ff4c80aceb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80097a0 tx=0x7ff4b802b820 comp rx=0 tx=0).stop 2024-04-25T19:44:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.246+0000 7ff4ce7cf640 1 -- 172.21.15.29:0/2037978752 shutdown_connections 2024-04-25T19:44:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.246+0000 7ff4ce7cf640 1 --2- 172.21.15.29:0/2037978752 >> v2:172.21.15.102:3300/0 conn(0x7ff4c8146c10 0x7ff4c8147020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.246+0000 7ff4ce7cf640 1 --2- 172.21.15.29:0/2037978752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4c80acaa0 0x7ff4c80aceb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.246+0000 7ff4ce7cf640 1 --2- 172.21.15.29:0/2037978752 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff4c8143e30 0x7ff4c8144240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.246+0000 7ff4ce7cf640 1 -- 172.21.15.29:0/2037978752 >> 172.21.15.29:0/2037978752 conn(0x7ff4c80aafc0 msgr2=0x7ff4c80ab3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.246+0000 7ff4ce7cf640 1 -- 172.21.15.29:0/2037978752 shutdown_connections 2024-04-25T19:44:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.246+0000 7ff4ce7cf640 1 -- 172.21.15.29:0/2037978752 wait complete. 2024-04-25T19:44:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.248+0000 7ff4ce7cf640 1 Processor -- start 2024-04-25T19:44:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.248+0000 7ff4ce7cf640 1 -- start start 2024-04-25T19:44:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.249+0000 7ff4ce7cf640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4c80acaa0 0x7ff4c8149800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.249+0000 7ff4ce7cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4c8143e30 0x7ff4c80afd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.249+0000 7ff4ce7cf640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff4c8146c10 0x7ff4c81e0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.249+0000 7ff4ce7cf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c814a590 con 0x7ff4c8143e30 2024-04-25T19:44:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.249+0000 7ff4ce7cf640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c814a410 con 0x7ff4c80acaa0 2024-04-25T19:44:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.249+0000 7ff4ce7cf640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c814a710 con 0x7ff4c8146c10 2024-04-25T19:44:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.249+0000 7ff4ccd6c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff4c8146c10 0x7ff4c81e0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.249+0000 7ff4c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4c8143e30 0x7ff4c80afd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.249+0000 7ff4c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4c8143e30 0x7ff4c80afd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39960/0 (socket says 172.21.15.29:39960) 2024-04-25T19:44:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.249+0000 7ff4c77fe640 1 -- 172.21.15.29:0/424153543 learned_addr learned my addr 172.21.15.29:0/424153543 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:23.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.249+0000 7ff4ccd6c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff4c8146c10 0x7ff4c81e0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:35338/0 (socket says 172.21.15.29:35338) 2024-04-25T19:44:23.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.249+0000 7ff4c7fff640 1 --2- 172.21.15.29:0/424153543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4c80acaa0 0x7ff4c8149800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.249+0000 7ff4ccd6c640 1 -- 172.21.15.29:0/424153543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4c80acaa0 msgr2=0x7ff4c8149800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:23.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.249+0000 7ff4ccd6c640 1 --2- 172.21.15.29:0/424153543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4c80acaa0 0x7ff4c8149800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.249+0000 7ff4ccd6c640 1 -- 172.21.15.29:0/424153543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4c8143e30 msgr2=0x7ff4c80afd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:23.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.249+0000 7ff4ccd6c640 1 --2- 172.21.15.29:0/424153543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4c8143e30 0x7ff4c80afd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.249+0000 7ff4ccd6c640 1 -- 172.21.15.29:0/424153543 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c81e0680 con 0x7ff4c8146c10 2024-04-25T19:44:23.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.249+0000 7ff4c77fe640 1 --2- 172.21.15.29:0/424153543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4c8143e30 0x7ff4c80afd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:23.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.249+0000 7ff4ccd6c640 1 --2- 172.21.15.29:0/424153543 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff4c8146c10 0x7ff4c81e0d80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000cc60 tx=0x7ff4c000a650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:23.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.250+0000 7ff4c4ff9640 1 -- 172.21.15.29:0/424153543 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c000f680 con 0x7ff4c8146c10 2024-04-25T19:44:23.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.250+0000 7ff4c4ff9640 1 -- 172.21.15.29:0/424153543 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c000f840 con 0x7ff4c8146c10 2024-04-25T19:44:23.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.250+0000 7ff4ce7cf640 1 -- 172.21.15.29:0/424153543 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4c81e12c0 con 0x7ff4c8146c10 2024-04-25T19:44:23.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.250+0000 7ff4c4ff9640 1 -- 172.21.15.29:0/424153543 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c000fc30 con 0x7ff4c8146c10 2024-04-25T19:44:23.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.250+0000 7ff4ce7cf640 1 -- 172.21.15.29:0/424153543 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4c81dc510 con 0x7ff4c8146c10 2024-04-25T19:44:23.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.251+0000 7ff4ce7cf640 1 -- 172.21.15.29:0/424153543 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4c81dd3f0 con 0x7ff4c8146c10 2024-04-25T19:44:23.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.252+0000 7ff4c4ff9640 1 -- 172.21.15.29:0/424153543 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7ff4c00213f0 con 0x7ff4c8146c10 2024-04-25T19:44:23.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.252+0000 7ff4c4ff9640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:23.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.253+0000 7ff4c4ff9640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:23.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.253+0000 7ff4c4ff9640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:23.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.253+0000 7ff4c4ff9640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:23.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.253+0000 7ff4c4ff9640 1 --2- 172.21.15.29:0/424153543 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7ff4b40ac560 0x7ff4b40aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.253+0000 7ff4c4ff9640 1 -- 172.21.15.29:0/424153543 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7ff4c00da1c0 con 0x7ff4c8146c10 2024-04-25T19:44:23.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.253+0000 7ff4c7fff640 1 --2- 172.21.15.29:0/424153543 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7ff4b40ac560 0x7ff4b40aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.254+0000 7ff4c7fff640 1 --2- 172.21.15.29:0/424153543 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7ff4b40ac560 0x7ff4b40aea70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc009a40 tx=0x7ff4bc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:23.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.257+0000 7ff4c4ff9640 1 -- 172.21.15.29:0/424153543 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff4c0014030 con 0x7ff4c8146c10 2024-04-25T19:44:23.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.479+0000 7ff4ce7cf640 1 -- 172.21.15.29:0/424153543 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4c80507b0 con 0x7ff4c8146c10 2024-04-25T19:44:23.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.480+0000 7ff4c4ff9640 1 -- 172.21.15.29:0/424153543 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+4351 (secure 0 0 0) 0x7ff4c00a4e30 con 0x7ff4c8146c10 2024-04-25T19:44:23.481 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:44:23.481 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6809/3582557422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3582557422},{"type":"v1","addr":"172.21.15.29:6809","nonce":3582557422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4234,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/1228868047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1228868047},{"type":"v1","addr":"172.21.15.102:6827","nonce":1228868047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:44:09.410562+0000","modified":"2024-04-25T19:44:12.425418+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6811/844243878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":844243878},{"type":"v1","addr":"172.21.15.29:6811","nonce":844243878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":1}]} 2024-04-25T19:44:23.481 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2024-04-25T19:44:23.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.481+0000 7ff4ce7cf640 10 mgrc shutdown 2024-04-25T19:44:23.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.481+0000 7ff4ce7cf640 1 -- 172.21.15.29:0/424153543 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7ff4b40ac560 msgr2=0x7ff4b40aea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:23.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.481+0000 7ff4ce7cf640 1 --2- 172.21.15.29:0/424153543 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7ff4b40ac560 0x7ff4b40aea70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc009a40 tx=0x7ff4bc008040 comp rx=0 tx=0).stop 2024-04-25T19:44:23.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.481+0000 7ff4ce7cf640 1 -- 172.21.15.29:0/424153543 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff4c8146c10 msgr2=0x7ff4c81e0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:23.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.481+0000 7ff4ce7cf640 1 --2- 172.21.15.29:0/424153543 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff4c8146c10 0x7ff4c81e0d80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000cc60 tx=0x7ff4c000a650 comp rx=0 tx=0).stop 2024-04-25T19:44:23.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.482+0000 7ff4ce7cf640 1 -- 172.21.15.29:0/424153543 shutdown_connections 2024-04-25T19:44:23.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.482+0000 7ff4ce7cf640 1 --2- 172.21.15.29:0/424153543 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7ff4b40ac560 0x7ff4b40aea70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.482+0000 7ff4ce7cf640 1 --2- 172.21.15.29:0/424153543 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff4c8146c10 0x7ff4c81e0d80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.482+0000 7ff4ce7cf640 1 --2- 172.21.15.29:0/424153543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4c8143e30 0x7ff4c80afd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.482+0000 7ff4ce7cf640 1 --2- 172.21.15.29:0/424153543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4c80acaa0 0x7ff4c8149800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.482+0000 7ff4ce7cf640 1 -- 172.21.15.29:0/424153543 >> 172.21.15.29:0/424153543 conn(0x7ff4c80aafc0 msgr2=0x7ff4c80684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:23.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.482+0000 7ff4ce7cf640 1 -- 172.21.15.29:0/424153543 shutdown_connections 2024-04-25T19:44:23.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.482+0000 7ff4ce7cf640 1 -- 172.21.15.29:0/424153543 wait complete. 2024-04-25T19:44:23.796 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T19:44:09.410562+0000', 'modified': '2024-04-25T19:44:12.425418+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4235}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4235': {'gid': 4235, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6811/844243878', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6810', 'nonce': 844243878}, {'type': 'v1', 'addr': '172.21.15.29:6811', 'nonce': 844243878}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4235, 'qdb_cluster': [4235]} 2024-04-25T19:44:23.796 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T19:44:23.797 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-25T19:44:23.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.910+0000 7f822d0b4640 1 Processor -- start 2024-04-25T19:44:23.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.910+0000 7f822d0b4640 1 -- start start 2024-04-25T19:44:23.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.911+0000 7f822d0b4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8228143e30 0x7f8228144240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.911+0000 7f822d0b4640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f82280acaa0 0x7f82280aceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.911+0000 7f822d0b4640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8228146c10 0x7f8228147020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.911+0000 7f822d0b4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f8228147560 con 0x7f8228143e30 2024-04-25T19:44:23.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.911+0000 7f822d0b4640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f82280a5a90 con 0x7f82280acaa0 2024-04-25T19:44:23.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.911+0000 7f822d0b4640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f8228147790 con 0x7f8228146c10 2024-04-25T19:44:23.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.911+0000 7f8226d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8228143e30 0x7f8228144240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.911+0000 7f822759e640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8228146c10 0x7f8228147020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.911+0000 7f8226d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8228143e30 0x7f8228144240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39976/0 (socket says 172.21.15.29:39976) 2024-04-25T19:44:23.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.911+0000 7f8226d9d640 1 -- 172.21.15.29:0/2521453178 learned_addr learned my addr 172.21.15.29:0/2521453178 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:23.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.911+0000 7f822659c640 1 --2- 172.21.15.29:0/2521453178 >> v2:172.21.15.102:3300/0 conn(0x7f82280acaa0 0x7f82280aceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.911+0000 7f8226d9d640 1 -- 172.21.15.29:0/2521453178 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8228146c10 msgr2=0x7f8228147020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:23.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.911+0000 7f8226d9d640 1 --2- 172.21.15.29:0/2521453178 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8228146c10 0x7f8228147020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.911+0000 7f8226d9d640 1 -- 172.21.15.29:0/2521453178 >> v2:172.21.15.102:3300/0 conn(0x7f82280acaa0 msgr2=0x7f82280aceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:23.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.911+0000 7f8226d9d640 1 --2- 172.21.15.29:0/2521453178 >> v2:172.21.15.102:3300/0 conn(0x7f82280acaa0 0x7f82280aceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.911+0000 7f8226d9d640 1 -- 172.21.15.29:0/2521453178 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8228147a50 con 0x7f8228143e30 2024-04-25T19:44:23.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.911+0000 7f822759e640 1 --2- 172.21.15.29:0/2521453178 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8228146c10 0x7f8228147020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:23.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.912+0000 7f8226d9d640 1 --2- 172.21.15.29:0/2521453178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8228143e30 0x7f8228144240 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f82280adbc0 tx=0x7f821802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=950d424cdec4f5b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:23.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.912+0000 7f8225d9b640 1 -- 172.21.15.29:0/2521453178 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821802ca80 con 0x7f8228143e30 2024-04-25T19:44:23.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.912+0000 7f8225d9b640 1 -- 172.21.15.29:0/2521453178 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f821802cc20 con 0x7f8228143e30 2024-04-25T19:44:23.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.912+0000 7f8225d9b640 1 -- 172.21.15.29:0/2521453178 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82180343f0 con 0x7f8228143e30 2024-04-25T19:44:23.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.913+0000 7f822d0b4640 1 -- 172.21.15.29:0/2521453178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8228143e30 msgr2=0x7f8228144240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:23.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.913+0000 7f822d0b4640 1 --2- 172.21.15.29:0/2521453178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8228143e30 0x7f8228144240 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f82280adbc0 tx=0x7f821802bee0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.913+0000 7f822d0b4640 1 -- 172.21.15.29:0/2521453178 shutdown_connections 2024-04-25T19:44:23.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.913+0000 7f822d0b4640 1 --2- 172.21.15.29:0/2521453178 >> v2:172.21.15.102:3300/0 conn(0x7f82280acaa0 0x7f82280aceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.913+0000 7f822d0b4640 1 --2- 172.21.15.29:0/2521453178 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8228146c10 0x7f8228147020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.913+0000 7f822d0b4640 1 --2- 172.21.15.29:0/2521453178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8228143e30 0x7f8228144240 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.913+0000 7f822d0b4640 1 -- 172.21.15.29:0/2521453178 >> 172.21.15.29:0/2521453178 conn(0x7f82280aafc0 msgr2=0x7f82280ab3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:23.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.913+0000 7f822d0b4640 1 -- 172.21.15.29:0/2521453178 shutdown_connections 2024-04-25T19:44:23.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.913+0000 7f822d0b4640 1 -- 172.21.15.29:0/2521453178 wait complete. 2024-04-25T19:44:23.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.915+0000 7f822d0b4640 1 Processor -- start 2024-04-25T19:44:23.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.915+0000 7f822d0b4640 1 -- start start 2024-04-25T19:44:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.916+0000 7f822d0b4640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82280acaa0 0x7f82281436b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.916+0000 7f822d0b4640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8228143e30 0x7f82280af690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.916+0000 7f822d0b4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8228146c10 0x7f82281e0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.916+0000 7f822d0b4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f822814a5d0 con 0x7f8228146c10 2024-04-25T19:44:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.916+0000 7f822d0b4640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f822814a450 con 0x7f82280acaa0 2024-04-25T19:44:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.916+0000 7f822d0b4640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f822814a750 con 0x7f8228143e30 2024-04-25T19:44:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.916+0000 7f822659c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8228143e30 0x7f82280af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.916+0000 7f822759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8228146c10 0x7f82281e0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.916+0000 7f822659c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8228143e30 0x7f82280af690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:35356/0 (socket says 172.21.15.29:35356) 2024-04-25T19:44:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.916+0000 7f822759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8228146c10 0x7f82281e0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39984/0 (socket says 172.21.15.29:39984) 2024-04-25T19:44:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.916+0000 7f822659c640 1 -- 172.21.15.29:0/1493713404 learned_addr learned my addr 172.21.15.29:0/1493713404 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:23.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.916+0000 7f8226d9d640 1 --2- 172.21.15.29:0/1493713404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82280acaa0 0x7f82281436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.916+0000 7f822659c640 1 -- 172.21.15.29:0/1493713404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82280acaa0 msgr2=0x7f82281436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:23.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.916+0000 7f822659c640 1 --2- 172.21.15.29:0/1493713404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82280acaa0 0x7f82281436b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.916+0000 7f822659c640 1 -- 172.21.15.29:0/1493713404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8228146c10 msgr2=0x7f82281e0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:23.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.916+0000 7f822659c640 1 --2- 172.21.15.29:0/1493713404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8228146c10 0x7f82281e0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.916+0000 7f822659c640 1 -- 172.21.15.29:0/1493713404 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82280afbd0 con 0x7f8228143e30 2024-04-25T19:44:23.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.916+0000 7f8226d9d640 1 --2- 172.21.15.29:0/1493713404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82280acaa0 0x7f82281436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:44:23.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.916+0000 7f822659c640 1 --2- 172.21.15.29:0/1493713404 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8228143e30 0x7f82280af690 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f821c00d800 tx=0x7f821c00dc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:23.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.917+0000 7f820f7fe640 1 -- 172.21.15.29:0/1493713404 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c018030 con 0x7f8228143e30 2024-04-25T19:44:23.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.917+0000 7f820f7fe640 1 -- 172.21.15.29:0/1493713404 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f821c009640 con 0x7f8228143e30 2024-04-25T19:44:23.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.917+0000 7f820f7fe640 1 -- 172.21.15.29:0/1493713404 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c009810 con 0x7f8228143e30 2024-04-25T19:44:23.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.917+0000 7f822d0b4640 1 -- 172.21.15.29:0/1493713404 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8228142ed0 con 0x7f8228143e30 2024-04-25T19:44:23.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.917+0000 7f822d0b4640 1 -- 172.21.15.29:0/1493713404 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f82281e1210 con 0x7f8228143e30 2024-04-25T19:44:23.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.919+0000 7f820f7fe640 1 -- 172.21.15.29:0/1493713404 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f821c01b070 con 0x7f8228143e30 2024-04-25T19:44:23.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.919+0000 7f820f7fe640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:44:23.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.920+0000 7f820f7fe640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:44:23.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.920+0000 7f820f7fe640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:23.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.920+0000 7f820f7fe640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:44:23.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.920+0000 7f820f7fe640 1 --2- 172.21.15.29:0/1493713404 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f82140ac560 0x7f82140aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.920+0000 7f8233e41740 1 Processor -- start 2024-04-25T19:44:23.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.920+0000 7f820f7fe640 1 -- 172.21.15.29:0/1493713404 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f821c0d9600 con 0x7f8228143e30 2024-04-25T19:44:23.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.920+0000 7f8233e41740 1 -- start start 2024-04-25T19:44:23.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.920+0000 7f8226d9d640 1 --2- 172.21.15.29:0/1493713404 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f82140ac560 0x7f82140aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.921+0000 7f8226d9d640 1 --2- 172.21.15.29:0/1493713404 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f82140ac560 0x7f82140aea70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8218035000 tx=0x7f8218030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:23.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.921+0000 7f8233e41740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564850e73270 0x56485120e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.921+0000 7f8233e41740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e4a3b0 0x5648512121e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.921+0000 7f8233e41740 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e52590 0x564851212720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.921+0000 7f8233e41740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5648510ff090 con 0x564850e4a3b0 2024-04-25T19:44:23.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.921+0000 7f8233e41740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x564851179580 con 0x564850e73270 2024-04-25T19:44:23.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.921+0000 7f8233e41740 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x564851115c10 con 0x564850e52590 2024-04-25T19:44:23.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.921+0000 7f822659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e4a3b0 0x5648512121e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.921+0000 7f822759e640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e52590 0x564851212720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.921+0000 7f822659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e4a3b0 0x5648512121e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39990/0 (socket says 172.21.15.29:39990) 2024-04-25T19:44:23.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.921+0000 7f822659c640 1 -- 172.21.15.29:0/1113811252 learned_addr learned my addr 172.21.15.29:0/1113811252 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:23.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.921+0000 7f8226d9d640 1 --2- 172.21.15.29:0/1113811252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564850e73270 0x56485120e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.921+0000 7f822759e640 1 -- 172.21.15.29:0/1113811252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564850e73270 msgr2=0x56485120e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:23.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.921+0000 7f822759e640 1 --2- 172.21.15.29:0/1113811252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564850e73270 0x56485120e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.921+0000 7f822759e640 1 -- 172.21.15.29:0/1113811252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e4a3b0 msgr2=0x5648512121e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:23.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.921+0000 7f822759e640 1 --2- 172.21.15.29:0/1113811252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e4a3b0 0x5648512121e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.921+0000 7f822759e640 1 -- 172.21.15.29:0/1113811252 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564850e8b2d0 con 0x564850e52590 2024-04-25T19:44:23.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.921+0000 7f822659c640 1 --2- 172.21.15.29:0/1113811252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e4a3b0 0x5648512121e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:23.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.922+0000 7f822759e640 1 --2- 172.21.15.29:0/1113811252 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e52590 0x564851212720 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8220009a30 tx=0x7f8220009e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e337e8d34a510132 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:23.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.922+0000 7f820cff9640 1 -- 172.21.15.29:0/1113811252 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8220011c30 con 0x564850e52590 2024-04-25T19:44:23.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.922+0000 7f820cff9640 1 -- 172.21.15.29:0/1113811252 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8220011dd0 con 0x564850e52590 2024-04-25T19:44:23.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.922+0000 7f820cff9640 1 -- 172.21.15.29:0/1113811252 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8220004240 con 0x564850e52590 2024-04-25T19:44:23.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.922+0000 7f8233e41740 1 -- 172.21.15.29:0/1113811252 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e52590 msgr2=0x564851212720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:23.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.922+0000 7f8233e41740 1 --2- 172.21.15.29:0/1113811252 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e52590 0x564851212720 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8220009a30 tx=0x7f8220009e30 comp rx=0 tx=0).stop 2024-04-25T19:44:23.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.923+0000 7f8233e41740 1 -- 172.21.15.29:0/1113811252 shutdown_connections 2024-04-25T19:44:23.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.923+0000 7f8233e41740 1 --2- 172.21.15.29:0/1113811252 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e52590 0x564851212720 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.923+0000 7f8233e41740 1 --2- 172.21.15.29:0/1113811252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e4a3b0 0x5648512121e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.923+0000 7f8233e41740 1 --2- 172.21.15.29:0/1113811252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564850e73270 0x56485120e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.923+0000 7f8233e41740 1 -- 172.21.15.29:0/1113811252 >> 172.21.15.29:0/1113811252 conn(0x564850e673c0 msgr2=0x5648512130f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:23.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.923+0000 7f8233e41740 1 -- 172.21.15.29:0/1113811252 shutdown_connections 2024-04-25T19:44:23.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.923+0000 7f8233e41740 1 -- 172.21.15.29:0/1113811252 wait complete. 2024-04-25T19:44:23.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.923+0000 7f8233e41740 1 -- start start 2024-04-25T19:44:23.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.923+0000 7f8233e41740 1 Processor -- start 2024-04-25T19:44:23.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.924+0000 7f8233e41740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e4a3b0 0x564851253170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.924+0000 7f8233e41740 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e52590 0x56485124e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.924+0000 7f8233e41740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564850e73270 0x56485124ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.924+0000 7f8233e41740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564851100a70 con 0x564850e4a3b0 2024-04-25T19:44:23.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.924+0000 7f8233e41740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x564851175540 con 0x564850e73270 2024-04-25T19:44:23.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.924+0000 7f8233e41740 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x56485122ab50 con 0x564850e52590 2024-04-25T19:44:23.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.924+0000 7f8233e41740 10 client.0 _ll_register_callbacks cb 0x56485111b7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:44:23.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.924+0000 7f822659c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e52590 0x56485124e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.924+0000 7f8226d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e4a3b0 0x564851253170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.924+0000 7f822659c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e52590 0x56485124e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:35384/0 (socket says 172.21.15.29:35384) 2024-04-25T19:44:23.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.924+0000 7f822659c640 1 -- 172.21.15.29:0/3527620058 learned_addr learned my addr 172.21.15.29:0/3527620058 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:23.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.925+0000 7f822759e640 1 --2- 172.21.15.29:0/3527620058 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564850e73270 0x56485124ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.925+0000 7f822659c640 1 -- 172.21.15.29:0/3527620058 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564850e73270 msgr2=0x56485124ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:23.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.925+0000 7f822659c640 1 --2- 172.21.15.29:0/3527620058 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564850e73270 0x56485124ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.925+0000 7f822659c640 1 -- 172.21.15.29:0/3527620058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e4a3b0 msgr2=0x564851253170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:23.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.925+0000 7f822659c640 1 --2- 172.21.15.29:0/3527620058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e4a3b0 0x564851253170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.925+0000 7f822659c640 1 -- 172.21.15.29:0/3527620058 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x564850f556b0 con 0x564850e52590 2024-04-25T19:44:23.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.925+0000 7f822759e640 1 --2- 172.21.15.29:0/3527620058 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564850e73270 0x56485124ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:44:23.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.925+0000 7f822659c640 1 --2- 172.21.15.29:0/3527620058 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e52590 0x56485124e8c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f821c009490 tx=0x7f821c0ac3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f901c02edfdd6c24 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:23.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.925+0000 7f8205ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3301/0 2024-04-25T19:44:23.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.925+0000 7f8205ffb640 1 -- 172.21.15.29:0/3527620058 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c0a7f20 con 0x564850e52590 2024-04-25T19:44:23.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.925+0000 7f8205ffb640 1 -- 172.21.15.29:0/3527620058 <== mon.2 v2:172.21.15.29:3301/0 2 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f821c0a1ac0 con 0x564850e52590 2024-04-25T19:44:23.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.925+0000 7f8233e41740 1 -- 172.21.15.29:0/3527620058 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8220011dd0 con 0x564850e52590 2024-04-25T19:44:23.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.925+0000 7f8205ffb640 1 -- 172.21.15.29:0/3527620058 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f821c0ac9d0 con 0x564850e52590 2024-04-25T19:44:23.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.925+0000 7f8233e41740 10 client.4505 fetch_fsmap learned FSMap version 8 2024-04-25T19:44:23.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.925+0000 7f8233e41740 1 -- 172.21.15.29:0/3527620058 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x564851237f20 con 0x564850e52590 2024-04-25T19:44:23.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.925+0000 7f8205ffb640 1 -- 172.21.15.29:0/3527620058 <== mon.2 v2:172.21.15.29:3301/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8220011dd0 con 0x564850e52590 2024-04-25T19:44:23.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.926+0000 7f8205ffb640 1 -- 172.21.15.29:0/3527620058 <== mon.2 v2:172.21.15.29:3301/0 5 ==== fsmap(e 8) ==== 2889+0+0 (secure 0 0 0) 0x7f821c025060 con 0x564850e52590 2024-04-25T19:44:23.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.926+0000 7f8233e41740 10 client.4505 fetch_fsmap finished waiting for FSMap version 8 2024-04-25T19:44:23.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.926+0000 7f8233e41740 10 client.4505 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T19:44:23.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.926+0000 7f8233e41740 20 client.4505 populate_metadata read hostname 'smithi029' 2024-04-25T19:44:23.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.926+0000 7f8233e41740 1 --2- 172.21.15.29:0/3527620058 >> [v2:172.21.15.29:6810/844243878,v1:172.21.15.29:6811/844243878] conn(0x564850e31e40 0x564851229550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.926+0000 7f8233e41740 4 client.4505 mds_command: new command op to 4235 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:44:23.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.926+0000 7f8233e41740 1 -- 172.21.15.29:0/3527620058 --> [v2:172.21.15.29:6810/844243878,v1:172.21.15.29:6811/844243878] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56485114c920 con 0x564850e31e40 2024-04-25T19:44:23.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.926+0000 7f8226d9d640 1 --2- 172.21.15.29:0/3527620058 >> [v2:172.21.15.29:6810/844243878,v1:172.21.15.29:6811/844243878] conn(0x564850e31e40 0x564851229550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.926+0000 7f8205ffb640 10 client.4505 ms_handle_connect on v2:172.21.15.29:6810/844243878 2024-04-25T19:44:23.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.926+0000 7f8226d9d640 1 --2- 172.21.15.29:0/3527620058 >> [v2:172.21.15.29:6810/844243878,v1:172.21.15.29:6811/844243878] conn(0x564850e31e40 0x564851229550 crc :-1 s=READY pgs=9 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=944eb950a7982670 server_cookie=e3bbd26270f6b20f in_seq=0 out_seq=0 2024-04-25T19:44:23.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.927+0000 7f8205ffb640 1 -- 172.21.15.29:0/3527620058 <== mds.0 v2:172.21.15.29:6810/844243878 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56485114c920 con 0x564850e31e40 2024-04-25T19:44:23.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.927+0000 7f8205ffb640 10 client.4505 handle_command_reply: tid=0 2024-04-25T19:44:23.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.927+0000 7f8233e41740 1 client.4505 shutdown 2024-04-25T19:44:23.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.927+0000 7f8233e41740 1 -- 172.21.15.29:0/3527620058 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e52590 msgr2=0x56485124e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:23.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.927+0000 7f8233e41740 1 --2- 172.21.15.29:0/3527620058 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e52590 0x56485124e8c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f821c009490 tx=0x7f821c0ac3d0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.927+0000 7f8233e41740 1 -- 172.21.15.29:0/3527620058 shutdown_connections 2024-04-25T19:44:23.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.927+0000 7f8233e41740 1 --2- 172.21.15.29:0/3527620058 >> [v2:172.21.15.29:6810/844243878,v1:172.21.15.29:6811/844243878] conn(0x564850e31e40 0x564851229550 crc :-1 s=READY pgs=9 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.927+0000 7f8233e41740 1 --2- 172.21.15.29:0/3527620058 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564850e73270 0x56485124ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.927+0000 7f8233e41740 1 --2- 172.21.15.29:0/3527620058 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e52590 0x56485124e8c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.927+0000 7f8205ffb640 0 client.4505 ms_handle_reset on v2:172.21.15.29:6810/844243878 2024-04-25T19:44:23.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.927+0000 7f8233e41740 1 --2- 172.21.15.29:0/3527620058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e4a3b0 0x564851253170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.927+0000 7f8233e41740 1 -- 172.21.15.29:0/3527620058 >> 172.21.15.29:0/3527620058 conn(0x564850e673c0 msgr2=0x56485120e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:23.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.927+0000 7f8233e41740 1 -- 172.21.15.29:0/3527620058 shutdown_connections 2024-04-25T19:44:23.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.927+0000 7f8233e41740 1 -- 172.21.15.29:0/3527620058 wait complete. 2024-04-25T19:44:23.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.927+0000 7f8233e41740 20 client.4505 trim_cache size 0 max 16384 2024-04-25T19:44:23.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.946+0000 7f8233e41740 1 Processor -- start 2024-04-25T19:44:23.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.946+0000 7f8233e41740 1 -- start start 2024-04-25T19:44:23.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.947+0000 7f8233e41740 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e4a3b0 0x56485124f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.947+0000 7f8233e41740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e52590 0x5648512644e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.947+0000 7f8233e41740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564850e73270 0x564851212d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.947+0000 7f8233e41740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f82281e9c90 con 0x564850e52590 2024-04-25T19:44:23.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.947+0000 7f8233e41740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f82281e88c0 con 0x564850e73270 2024-04-25T19:44:23.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.947+0000 7f8233e41740 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f82281e8d90 con 0x564850e4a3b0 2024-04-25T19:44:23.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.947+0000 7f822659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e52590 0x5648512644e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.947+0000 7f8226d9d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e4a3b0 0x56485124f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.947+0000 7f822659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e52590 0x5648512644e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40012/0 (socket says 172.21.15.29:40012) 2024-04-25T19:44:23.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.947+0000 7f8226d9d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e4a3b0 0x56485124f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:35388/0 (socket says 172.21.15.29:35388) 2024-04-25T19:44:23.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.947+0000 7f822659c640 1 -- 172.21.15.29:0/4140943954 learned_addr learned my addr 172.21.15.29:0/4140943954 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:23.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.947+0000 7f822759e640 1 --2- 172.21.15.29:0/4140943954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564850e73270 0x564851212d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.947+0000 7f822659c640 1 -- 172.21.15.29:0/4140943954 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e4a3b0 msgr2=0x56485124f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:23.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.947+0000 7f822659c640 1 --2- 172.21.15.29:0/4140943954 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e4a3b0 0x56485124f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.947+0000 7f822659c640 1 -- 172.21.15.29:0/4140943954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564850e73270 msgr2=0x564851212d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:23.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.947+0000 7f822659c640 1 --2- 172.21.15.29:0/4140943954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564850e73270 0x564851212d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.947+0000 7f822659c640 1 -- 172.21.15.29:0/4140943954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648512505a0 con 0x564850e52590 2024-04-25T19:44:23.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.947+0000 7f8226d9d640 1 --2- 172.21.15.29:0/4140943954 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e4a3b0 0x56485124f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:23.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.948+0000 7f822659c640 1 --2- 172.21.15.29:0/4140943954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e52590 0x5648512644e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f821c0d9850 tx=0x7f821c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a7fca490d763e1f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:23.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.948+0000 7f8207fff640 1 -- 172.21.15.29:0/4140943954 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c0a7f20 con 0x564850e52590 2024-04-25T19:44:23.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.948+0000 7f8207fff640 1 -- 172.21.15.29:0/4140943954 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f821c0a1ac0 con 0x564850e52590 2024-04-25T19:44:23.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.948+0000 7f8207fff640 1 -- 172.21.15.29:0/4140943954 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c0c50b0 con 0x564850e52590 2024-04-25T19:44:23.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.948+0000 7f8233e41740 1 -- 172.21.15.29:0/4140943954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e52590 msgr2=0x5648512644e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:23.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.948+0000 7f8233e41740 1 --2- 172.21.15.29:0/4140943954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e52590 0x5648512644e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f821c0d9850 tx=0x7f821c015000 comp rx=0 tx=0).stop 2024-04-25T19:44:23.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.948+0000 7f8233e41740 1 -- 172.21.15.29:0/4140943954 shutdown_connections 2024-04-25T19:44:23.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.948+0000 7f8233e41740 1 --2- 172.21.15.29:0/4140943954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564850e73270 0x564851212d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.948+0000 7f8233e41740 1 --2- 172.21.15.29:0/4140943954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e52590 0x5648512644e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.948+0000 7f8233e41740 1 --2- 172.21.15.29:0/4140943954 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e4a3b0 0x56485124f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.948+0000 7f8233e41740 1 -- 172.21.15.29:0/4140943954 >> 172.21.15.29:0/4140943954 conn(0x564850e673c0 msgr2=0x56485122a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:23.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.948+0000 7f8233e41740 1 -- 172.21.15.29:0/4140943954 shutdown_connections 2024-04-25T19:44:23.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.948+0000 7f8233e41740 1 -- 172.21.15.29:0/4140943954 wait complete. 2024-04-25T19:44:23.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.949+0000 7f8233e41740 1 -- start start 2024-04-25T19:44:23.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.949+0000 7f8233e41740 1 Processor -- start 2024-04-25T19:44:23.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.950+0000 7f8233e41740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564850e4a3b0 0x5648512bbec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.950+0000 7f8233e41740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e52590 0x5648512bc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.950+0000 7f8233e41740 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e73270 0x564851234040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.950+0000 7f8233e41740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564851232470 con 0x564850e52590 2024-04-25T19:44:23.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.950+0000 7f8233e41740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x564851238260 con 0x564850e4a3b0 2024-04-25T19:44:23.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.950+0000 7f8233e41740 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x564851232a90 con 0x564850e73270 2024-04-25T19:44:23.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.950+0000 7f8233e41740 10 client.0 _ll_register_callbacks cb 0x7f821c0a35b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:44:23.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.950+0000 7f822659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e52590 0x5648512bc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.950+0000 7f822659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e52590 0x5648512bc400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40024/0 (socket says 172.21.15.29:40024) 2024-04-25T19:44:23.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.950+0000 7f822659c640 1 -- 172.21.15.29:0/1092859320 learned_addr learned my addr 172.21.15.29:0/1092859320 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:44:23.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.950+0000 7f822759e640 1 --2- 172.21.15.29:0/1092859320 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e73270 0x564851234040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.950+0000 7f8226d9d640 1 --2- 172.21.15.29:0/1092859320 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564850e4a3b0 0x5648512bbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.950+0000 7f822659c640 1 -- 172.21.15.29:0/1092859320 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e73270 msgr2=0x564851234040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:23.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f822659c640 1 --2- 172.21.15.29:0/1092859320 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e73270 0x564851234040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f822659c640 1 -- 172.21.15.29:0/1092859320 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564850e4a3b0 msgr2=0x5648512bbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:44:23.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f822659c640 1 --2- 172.21.15.29:0/1092859320 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564850e4a3b0 0x5648512bbec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f822659c640 1 -- 172.21.15.29:0/1092859320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564851240d50 con 0x564850e52590 2024-04-25T19:44:23.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f822759e640 1 --2- 172.21.15.29:0/1092859320 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e73270 0x564851234040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:44:23.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f822659c640 1 --2- 172.21.15.29:0/1092859320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e52590 0x5648512bc400 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f821c0b9fd0 tx=0x7f821c0d7f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3ba6b0ef8aef898 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:44:23.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f8205ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-25T19:44:23.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f8205ffb640 1 -- 172.21.15.29:0/1092859320 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c087060 con 0x564850e52590 2024-04-25T19:44:23.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f8205ffb640 1 -- 172.21.15.29:0/1092859320 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(20..20 src has 1..20) ==== 4334+0+0 (secure 0 0 0) 0x7f821c0c52b0 con 0x564850e52590 2024-04-25T19:44:23.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f8233e41740 1 -- 172.21.15.29:0/1092859320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f821c0a1ac0 con 0x564850e52590 2024-04-25T19:44:23.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f8205ffb640 1 -- 172.21.15.29:0/1092859320 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f821c0a7f20 con 0x564850e52590 2024-04-25T19:44:23.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f8233e41740 10 client.4599 fetch_fsmap learned FSMap version 8 2024-04-25T19:44:23.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f8233e41740 1 -- 172.21.15.29:0/1092859320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x564851213810 con 0x564850e52590 2024-04-25T19:44:23.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f8205ffb640 1 -- 172.21.15.29:0/1092859320 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f821c0a1ac0 con 0x564850e52590 2024-04-25T19:44:23.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f8205ffb640 1 -- 172.21.15.29:0/1092859320 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 8) ==== 2889+0+0 (secure 0 0 0) 0x7f821c013050 con 0x564850e52590 2024-04-25T19:44:23.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f8233e41740 10 client.4599 fetch_fsmap finished waiting for FSMap version 8 2024-04-25T19:44:23.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f8233e41740 10 client.4599 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T19:44:23.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f8233e41740 20 client.4599 populate_metadata read hostname 'smithi029' 2024-04-25T19:44:23.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f8233e41740 1 --2- 172.21.15.29:0/1092859320 >> [v2:172.21.15.29:6810/844243878,v1:172.21.15.29:6811/844243878] conn(0x564851234580 0x564851234990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:44:23.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f8233e41740 4 client.4599 mds_command: new command op to 4235 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T19:44:23.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f8233e41740 1 -- 172.21.15.29:0/1092859320 --> [v2:172.21.15.29:6810/844243878,v1:172.21.15.29:6811/844243878] -- command(tid 0: {"prefix": "status"}) -- 0x564851187190 con 0x564851234580 2024-04-25T19:44:23.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.951+0000 7f8226d9d640 1 --2- 172.21.15.29:0/1092859320 >> [v2:172.21.15.29:6810/844243878,v1:172.21.15.29:6811/844243878] conn(0x564851234580 0x564851234990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:44:23.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.952+0000 7f8205ffb640 10 client.4599 ms_handle_connect on v2:172.21.15.29:6810/844243878 2024-04-25T19:44:23.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.952+0000 7f8226d9d640 1 --2- 172.21.15.29:0/1092859320 >> [v2:172.21.15.29:6810/844243878,v1:172.21.15.29:6811/844243878] conn(0x564851234580 0x564851234990 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6e03b21bb1698ab server_cookie=dfa30b90941000cd in_seq=0 out_seq=0 2024-04-25T19:44:23.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.952+0000 7f8205ffb640 1 -- 172.21.15.29:0/1092859320 <== mds.0 v2:172.21.15.29:6810/844243878 1 ==== command_reply(tid 0: 0 ) ==== 8+0+317 (crc 0 0 0) 0x564851187190 con 0x564851234580 2024-04-25T19:44:23.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.952+0000 7f8205ffb640 10 client.4599 handle_command_reply: tid=0 2024-04-25T19:44:23.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.952+0000 7f8233e41740 1 client.4599 shutdown 2024-04-25T19:44:23.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.952+0000 7f8233e41740 1 -- 172.21.15.29:0/1092859320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e52590 msgr2=0x5648512bc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:23.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.952+0000 7f8233e41740 1 --2- 172.21.15.29:0/1092859320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e52590 0x5648512bc400 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f821c0b9fd0 tx=0x7f821c0d7f60 comp rx=0 tx=0).stop 2024-04-25T19:44:23.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.952+0000 7f8233e41740 1 -- 172.21.15.29:0/1092859320 shutdown_connections 2024-04-25T19:44:23.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.952+0000 7f8233e41740 1 --2- 172.21.15.29:0/1092859320 >> [v2:172.21.15.29:6810/844243878,v1:172.21.15.29:6811/844243878] conn(0x564851234580 0x564851234990 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.952+0000 7f8233e41740 1 --2- 172.21.15.29:0/1092859320 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x564850e73270 0x564851234040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.952+0000 7f8233e41740 1 --2- 172.21.15.29:0/1092859320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564850e52590 0x5648512bc400 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.952+0000 7f8205ffb640 0 client.4599 ms_handle_reset on v2:172.21.15.29:6810/844243878 2024-04-25T19:44:23.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.952+0000 7f8233e41740 1 --2- 172.21.15.29:0/1092859320 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564850e4a3b0 0x5648512bbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.952+0000 7f8233e41740 1 -- 172.21.15.29:0/1092859320 >> 172.21.15.29:0/1092859320 conn(0x564850e673c0 msgr2=0x56485122a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:23.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.952+0000 7f8233e41740 1 -- 172.21.15.29:0/1092859320 shutdown_connections 2024-04-25T19:44:23.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.952+0000 7f8233e41740 1 -- 172.21.15.29:0/1092859320 wait complete. 2024-04-25T19:44:23.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.952+0000 7f8233e41740 20 client.4599 trim_cache size 0 max 16384 2024-04-25T19:44:23.961 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-04-25T19:44:23.961 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "1b3c39b5-e787-409e-b0bd-f4f491c89d72", 2024-04-25T19:44:23.961 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2024-04-25T19:44:23.961 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 4235, 2024-04-25T19:44:23.961 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-04-25T19:44:23.961 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-04-25T19:44:23.961 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-04-25T19:44:23.961 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 13.927400707, 2024-04-25T19:44:23.961 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 8, 2024-04-25T19:44:23.962 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 20, 2024-04-25T19:44:23.962 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 20, 2024-04-25T19:44:23.962 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 18.94231426 2024-04-25T19:44:23.962 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-04-25T19:44:23.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.953+0000 7f822d0b4640 10 mgrc shutdown 2024-04-25T19:44:23.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.953+0000 7f822d0b4640 1 -- 172.21.15.29:0/1493713404 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f82140ac560 msgr2=0x7f82140aea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:23.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.953+0000 7f822d0b4640 1 --2- 172.21.15.29:0/1493713404 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f82140ac560 0x7f82140aea70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8218035000 tx=0x7f8218030000 comp rx=0 tx=0).stop 2024-04-25T19:44:23.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.953+0000 7f822d0b4640 1 -- 172.21.15.29:0/1493713404 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8228143e30 msgr2=0x7f82280af690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:44:23.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.953+0000 7f822d0b4640 1 --2- 172.21.15.29:0/1493713404 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8228143e30 0x7f82280af690 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f821c00d800 tx=0x7f821c00dc00 comp rx=0 tx=0).stop 2024-04-25T19:44:23.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.953+0000 7f822d0b4640 1 -- 172.21.15.29:0/1493713404 shutdown_connections 2024-04-25T19:44:23.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.953+0000 7f822d0b4640 1 --2- 172.21.15.29:0/1493713404 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f82140ac560 0x7f82140aea70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.953+0000 7f822d0b4640 1 --2- 172.21.15.29:0/1493713404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8228146c10 0x7f82281e0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.953+0000 7f822d0b4640 1 --2- 172.21.15.29:0/1493713404 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8228143e30 0x7f82280af690 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.953+0000 7f822d0b4640 1 --2- 172.21.15.29:0/1493713404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82280acaa0 0x7f82281436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:44:23.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.953+0000 7f822d0b4640 1 -- 172.21.15.29:0/1493713404 >> 172.21.15.29:0/1493713404 conn(0x7f82280aafc0 msgr2=0x7f82280684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:44:23.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.953+0000 7f822d0b4640 1 -- 172.21.15.29:0/1493713404 shutdown_connections 2024-04-25T19:44:23.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:44:23.953+0000 7f822d0b4640 1 -- 172.21.15.29:0/1493713404 wait complete. 2024-04-25T19:44:23.965 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T19:44:23.989 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T19:44:23.989 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T19:44:23.989 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T19:44:23.989 INFO:teuthology.run_tasks:Running task workunit... 2024-04-25T19:44:24.005 INFO:tasks.workunit:Pulling workunits from ref 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T19:44:24.005 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-25T19:44:24.006 DEBUG:teuthology.orchestra.run.smithi029:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-25T19:44:24.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:44:24.021 INFO:teuthology.orchestra.run.smithi029.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-25T19:44:24.021 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-25T19:44:24.077 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-25T19:44:24.077 DEBUG:teuthology.orchestra.run.smithi029:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-25T19:44:24.133 INFO:tasks.workunit:timeout=3h 2024-04-25T19:44:24.134 INFO:tasks.workunit:cleanup=True 2024-04-25T19:44:24.135 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T19:44:24.189 INFO:tasks.workunit.client.0.smithi029.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-25T19:44:28.176 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:44:28.179 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:44:58.300 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:44:58.303 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:45:28.414 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:45:28.417 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:45:47.687 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files: 90% (11090/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-25T19:45:48.473 INFO:tasks.workunit.client.0.smithi029.stderr:Note: switching to '994e2f20102a20fa9a27966c14b625cc01884df8'. 2024-04-25T19:45:48.473 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-25T19:45:48.473 INFO:tasks.workunit.client.0.smithi029.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-25T19:45:48.473 INFO:tasks.workunit.client.0.smithi029.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-25T19:45:48.473 INFO:tasks.workunit.client.0.smithi029.stderr:state without impacting any branches by switching back to a branch. 2024-04-25T19:45:48.473 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-25T19:45:48.473 INFO:tasks.workunit.client.0.smithi029.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-25T19:45:48.473 INFO:tasks.workunit.client.0.smithi029.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-25T19:45:48.474 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-25T19:45:48.474 INFO:tasks.workunit.client.0.smithi029.stderr: git switch -c 2024-04-25T19:45:48.474 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-25T19:45:48.474 INFO:tasks.workunit.client.0.smithi029.stderr:Or undo this operation with: 2024-04-25T19:45:48.474 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-25T19:45:48.474 INFO:tasks.workunit.client.0.smithi029.stderr: git switch - 2024-04-25T19:45:48.474 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-25T19:45:48.474 INFO:tasks.workunit.client.0.smithi029.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-25T19:45:48.474 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-25T19:45:48.474 INFO:tasks.workunit.client.0.smithi029.stderr:HEAD is now at 994e2f20102 Merge PR #56810 into wip-pdonnell-testing-20240425.015853-debug 2024-04-25T19:45:48.479 DEBUG:teuthology.orchestra.run.smithi029:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-25T19:45:48.514 INFO:tasks.workunit.client.0.smithi029.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-25T19:45:48.517 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T19:45:48.517 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-25T19:45:49.379 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-25T19:45:49.467 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-25T19:45:49.509 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T19:45:49.510 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T19:45:49.510 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-25T19:45:49.550 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T19:45:49.555 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-25T19:45:49.555 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-25T19:45:49.611 INFO:tasks.workunit:Running workunits matching mgr/test_localpool.sh on client.0... 2024-04-25T19:45:49.612 INFO:tasks.workunit:Running workunit mgr/test_localpool.sh... 2024-04-25T19:45:49.612 DEBUG:teuthology.orchestra.run.smithi029:workunit test mgr/test_localpool.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=994e2f20102a20fa9a27966c14b625cc01884df8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mgr/test_localpool.sh 2024-04-25T19:45:49.670 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph config set mgr mgr/localpool/subtree host 2024-04-25T19:45:49.762 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.762+0000 7f57ef5bd640 1 Processor -- start 2024-04-25T19:45:49.762 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.762+0000 7f57ef5bd640 1 -- start start 2024-04-25T19:45:49.765 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.763+0000 7f57ef5bd640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f57e8070840 0x7f57e806d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:49.765 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.763+0000 7f57ef5bd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f57e814f650 0x7f57e8151a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:49.765 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.763+0000 7f57ef5bd640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57e806dce0 con 0x7f57e81479f0 2024-04-25T19:45:49.765 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.763+0000 7f57ef5bd640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f57e8070c50 con 0x7f57e814f650 2024-04-25T19:45:49.765 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.763+0000 7f57ef5bd640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f57e8146e70 con 0x7f57e8070840 2024-04-25T19:45:49.765 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.763+0000 7f57ed359640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f57e81479f0 0x7f57e80abc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50834/0 (socket says 172.21.15.29:50834) 2024-04-25T19:45:49.765 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.763+0000 7f57ed359640 1 -- 172.21.15.29:0/147001006 learned_addr learned my addr 172.21.15.29:0/147001006 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:45:49.765 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.764+0000 7f57edb5a640 1 --2- 172.21.15.29:0/147001006 >> v2:172.21.15.29:3300/0 conn(0x7f57e814f650 0x7f57e8151a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:49.765 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.764+0000 7f57ecb58640 1 --2- 172.21.15.29:0/147001006 >> v2:172.21.15.102:3300/0 conn(0x7f57e8070840 0x7f57e806d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:49.766 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.764+0000 7f57e7fff640 1 -- 172.21.15.29:0/147001006 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3728031858 0 0) 0x7f57e806dce0 con 0x7f57e81479f0 2024-04-25T19:45:49.766 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.764+0000 7f57e7fff640 1 -- 172.21.15.29:0/147001006 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57d0003520 con 0x7f57e81479f0 2024-04-25T19:45:49.766 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.764+0000 7f57edb5a640 1 -- 172.21.15.29:0/147001006 >> v2:172.21.15.102:3300/0 conn(0x7f57e8070840 msgr2=0x7f57e806d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:45:49.766 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.764+0000 7f57edb5a640 1 --2- 172.21.15.29:0/147001006 >> v2:172.21.15.102:3300/0 conn(0x7f57e8070840 0x7f57e806d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:49.766 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.764+0000 7f57edb5a640 1 -- 172.21.15.29:0/147001006 >> v1:172.21.15.29:6789/0 conn(0x7f57e81479f0 legacy=0x7f57e80abc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:49.766 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.764+0000 7f57edb5a640 1 -- 172.21.15.29:0/147001006 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57e80aea70 con 0x7f57e814f650 2024-04-25T19:45:49.766 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.765+0000 7f57edb5a640 1 --2- 172.21.15.29:0/147001006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57e814f650 0x7f57e8151a70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f57e806cbf0 tx=0x7f57d802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be8ae380269520e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:45:49.766 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.765+0000 7f57e7fff640 1 -- 172.21.15.29:0/147001006 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d8039070 con 0x7f57e814f650 2024-04-25T19:45:49.766 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.765+0000 7f57e7fff640 1 -- 172.21.15.29:0/147001006 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57d802cc80 con 0x7f57e814f650 2024-04-25T19:45:49.766 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.765+0000 7f57e7fff640 1 -- 172.21.15.29:0/147001006 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d8034660 con 0x7f57e814f650 2024-04-25T19:45:49.767 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.765+0000 7f57ef5bd640 1 -- 172.21.15.29:0/147001006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57e814f650 msgr2=0x7f57e8151a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:49.767 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.765+0000 7f57ef5bd640 1 --2- 172.21.15.29:0/147001006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57e814f650 0x7f57e8151a70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f57e806cbf0 tx=0x7f57d802bed0 comp rx=0 tx=0).stop 2024-04-25T19:45:49.767 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.765+0000 7f57ef5bd640 1 -- 172.21.15.29:0/147001006 shutdown_connections 2024-04-25T19:45:49.768 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.765+0000 7f57ef5bd640 1 --2- 172.21.15.29:0/147001006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57e814f650 0x7f57e8151a70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:49.768 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.765+0000 7f57ef5bd640 1 --2- 172.21.15.29:0/147001006 >> v2:172.21.15.102:3300/0 conn(0x7f57e8070840 0x7f57e806d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:49.768 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.765+0000 7f57ef5bd640 1 -- 172.21.15.29:0/147001006 >> 172.21.15.29:0/147001006 conn(0x7f57e80bc770 msgr2=0x7f57e80bcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:45:49.768 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.766+0000 7f57ef5bd640 1 -- 172.21.15.29:0/147001006 shutdown_connections 2024-04-25T19:45:49.768 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.766+0000 7f57ef5bd640 1 -- 172.21.15.29:0/147001006 wait complete. 2024-04-25T19:45:49.768 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.767+0000 7f57ef5bd640 1 Processor -- start 2024-04-25T19:45:49.768 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.767+0000 7f57ef5bd640 1 -- start start 2024-04-25T19:45:49.770 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.768+0000 7f57ef5bd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57e8070840 0x7f57e809f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:49.771 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.768+0000 7f57ef5bd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57e81479f0 0x7f57e813e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:49.771 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.768+0000 7f57ef5bd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f57e814f650 0x7f57e80b8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:49.771 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.768+0000 7f57ef5bd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f57e806dac0 con 0x7f57e8070840 2024-04-25T19:45:49.771 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.768+0000 7f57ef5bd640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f57e806d940 con 0x7f57e81479f0 2024-04-25T19:45:49.771 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.768+0000 7f57ef5bd640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f57e806e870 con 0x7f57e814f650 2024-04-25T19:45:49.771 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.768+0000 7f57ed359640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57e8070840 0x7f57e809f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:49.771 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.768+0000 7f57ed359640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57e8070840 0x7f57e809f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45808/0 (socket says 172.21.15.29:45808) 2024-04-25T19:45:49.771 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.768+0000 7f57ed359640 1 -- 172.21.15.29:0/2555431395 learned_addr learned my addr 172.21.15.29:0/2555431395 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:45:49.771 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.768+0000 7f57edb5a640 1 --2- 172.21.15.29:0/2555431395 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f57e814f650 0x7f57e80b8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:49.771 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.768+0000 7f57ecb58640 1 --2- 172.21.15.29:0/2555431395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57e81479f0 0x7f57e813e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:49.771 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.768+0000 7f57ed359640 1 -- 172.21.15.29:0/2555431395 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f57e814f650 msgr2=0x7f57e80b8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:49.771 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.768+0000 7f57ed359640 1 --2- 172.21.15.29:0/2555431395 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f57e814f650 0x7f57e80b8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:49.771 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.768+0000 7f57ed359640 1 -- 172.21.15.29:0/2555431395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57e81479f0 msgr2=0x7f57e813e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:49.771 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.768+0000 7f57ed359640 1 --2- 172.21.15.29:0/2555431395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57e81479f0 0x7f57e813e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:49.771 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.768+0000 7f57ed359640 1 -- 172.21.15.29:0/2555431395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57e813e240 con 0x7f57e8070840 2024-04-25T19:45:49.772 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.768+0000 7f57ecb58640 1 --2- 172.21.15.29:0/2555431395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57e81479f0 0x7f57e813e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:45:49.772 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.768+0000 7f57edb5a640 1 --2- 172.21.15.29:0/2555431395 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f57e814f650 0x7f57e80b8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:45:49.772 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.769+0000 7f57ed359640 1 --2- 172.21.15.29:0/2555431395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57e8070840 0x7f57e809f040 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f57e000c390 tx=0x7f57e000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:45:49.772 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.769+0000 7f57e5ffb640 1 -- 172.21.15.29:0/2555431395 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e0017070 con 0x7f57e8070840 2024-04-25T19:45:49.772 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.769+0000 7f57e5ffb640 1 -- 172.21.15.29:0/2555431395 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57e0005d00 con 0x7f57e8070840 2024-04-25T19:45:49.772 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.769+0000 7f57e5ffb640 1 -- 172.21.15.29:0/2555431395 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e0007310 con 0x7f57e8070840 2024-04-25T19:45:49.772 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.769+0000 7f57ef5bd640 1 -- 172.21.15.29:0/2555431395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57e8142a00 con 0x7f57e8070840 2024-04-25T19:45:49.772 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.769+0000 7f57ef5bd640 1 -- 172.21.15.29:0/2555431395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57e813eca0 con 0x7f57e8070840 2024-04-25T19:45:49.772 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.770+0000 7f57ef5bd640 1 -- 172.21.15.29:0/2555431395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57e814a230 con 0x7f57e8070840 2024-04-25T19:45:49.773 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.771+0000 7f57e5ffb640 1 -- 172.21.15.29:0/2555431395 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f57e001a070 con 0x7f57e8070840 2024-04-25T19:45:49.773 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.771+0000 7f57e5ffb640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:45:49.773 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.772+0000 7f57e5ffb640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:45:49.773 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.772+0000 7f57e5ffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:45:49.773 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.772+0000 7f57e5ffb640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:45:49.773 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.772+0000 7f57e5ffb640 1 --2- 172.21.15.29:0/2555431395 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f57d40ac530 0x7f57d40aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:49.773 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.772+0000 7f57e5ffb640 1 -- 172.21.15.29:0/2555431395 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 4450+0+0 (secure 0 0 0) 0x7f57e00d8780 con 0x7f57e8070840 2024-04-25T19:45:49.774 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.772+0000 7f57ecb58640 1 --2- 172.21.15.29:0/2555431395 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f57d40ac530 0x7f57d40aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:49.774 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.773+0000 7f57ecb58640 1 --2- 172.21.15.29:0/2555431395 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f57d40ac530 0x7f57d40aea40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f57e8074810 tx=0x7f57dc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:45:49.776 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.776+0000 7f57e5ffb640 1 -- 172.21.15.29:0/2555431395 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f57e004eb80 con 0x7f57e8070840 2024-04-25T19:45:49.962 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.962+0000 7f57ef5bd640 1 -- 172.21.15.29:0/2555431395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/localpool/subtree}] v 0) -- 0x7f57e80692d0 con 0x7f57e8070840 2024-04-25T19:45:49.967 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:49.966+0000 7f57e5ffb640 1 -- 172.21.15.29:0/2555431395 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/localpool/subtree}]=0 v6)=0 v6) ==== 122+0+0 (secure 0 0 0) 0x7f57e00c2e20 con 0x7f57e8070840 2024-04-25T19:45:50.076 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.075+0000 7f57ef5bd640 1 -- 172.21.15.29:0/2555431395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/localpool/subtree}] v 0) -- 0x7f57e80ae850 con 0x7f57e8070840 2024-04-25T19:45:50.076 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.076+0000 7f57e5ffb640 1 -- 172.21.15.29:0/2555431395 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mgr/localpool/subtree}]=0 v6)=0 v6) ==== 122+0+0 (secure 0 0 0) 0x7f57e00a3370 con 0x7f57e8070840 2024-04-25T19:45:50.079 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.078+0000 7f57ef5bd640 10 mgrc shutdown 2024-04-25T19:45:50.079 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.078+0000 7f57ef5bd640 1 -- 172.21.15.29:0/2555431395 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f57d40ac530 msgr2=0x7f57d40aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:50.079 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.078+0000 7f57ef5bd640 1 --2- 172.21.15.29:0/2555431395 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f57d40ac530 0x7f57d40aea40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f57e8074810 tx=0x7f57dc008040 comp rx=0 tx=0).stop 2024-04-25T19:45:50.079 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.078+0000 7f57ef5bd640 1 -- 172.21.15.29:0/2555431395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57e8070840 msgr2=0x7f57e809f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:50.080 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.078+0000 7f57ef5bd640 1 --2- 172.21.15.29:0/2555431395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57e8070840 0x7f57e809f040 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f57e000c390 tx=0x7f57e000c850 comp rx=0 tx=0).stop 2024-04-25T19:45:50.080 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.078+0000 7f57ef5bd640 1 -- 172.21.15.29:0/2555431395 shutdown_connections 2024-04-25T19:45:50.080 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.078+0000 7f57ef5bd640 1 --2- 172.21.15.29:0/2555431395 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f57d40ac530 0x7f57d40aea40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:50.080 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.078+0000 7f57ef5bd640 1 --2- 172.21.15.29:0/2555431395 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f57e814f650 0x7f57e80b8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:50.080 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.078+0000 7f57ef5bd640 1 --2- 172.21.15.29:0/2555431395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57e81479f0 0x7f57e813e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:50.080 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.078+0000 7f57ef5bd640 1 --2- 172.21.15.29:0/2555431395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57e8070840 0x7f57e809f040 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:50.081 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.078+0000 7f57ef5bd640 1 -- 172.21.15.29:0/2555431395 >> 172.21.15.29:0/2555431395 conn(0x7f57e80bc770 msgr2=0x7f57e8070dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:45:50.081 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.078+0000 7f57ef5bd640 1 -- 172.21.15.29:0/2555431395 shutdown_connections 2024-04-25T19:45:50.081 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.078+0000 7f57ef5bd640 1 -- 172.21.15.29:0/2555431395 wait complete. 2024-04-25T19:45:50.088 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph config set mgr mgr/localpool/failure_domain osd 2024-04-25T19:45:50.182 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.181+0000 7f0faa8a2640 1 Processor -- start 2024-04-25T19:45:50.182 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.181+0000 7f0faa8a2640 1 -- start start 2024-04-25T19:45:50.185 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.182+0000 7f0faa8a2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0fa41479f0 0x7f0fa4070840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:50.185 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.182+0000 7f0faa8a2640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0fa40abc70 0x7f0fa406d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:50.185 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.182+0000 7f0faa8a2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fa406dce0 con 0x7f0fa40ac900 2024-04-25T19:45:50.185 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.182+0000 7f0faa8a2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f0fa4146e70 con 0x7f0fa41479f0 2024-04-25T19:45:50.185 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.182+0000 7f0faa8a2640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f0fa40aea70 con 0x7f0fa40abc70 2024-04-25T19:45:50.185 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.182+0000 7f0fa3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0fa41479f0 0x7f0fa4070840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:50.185 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.182+0000 7f0fa8e3f640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0fa40ac900 0x7f0fa40acd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50850/0 (socket says 172.21.15.29:50850) 2024-04-25T19:45:50.185 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.182+0000 7f0fa8e3f640 1 -- 172.21.15.29:0/1243916239 learned_addr learned my addr 172.21.15.29:0/1243916239 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:45:50.186 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.183+0000 7f0fa37fe640 1 --2- 172.21.15.29:0/1243916239 >> v2:172.21.15.102:3300/0 conn(0x7f0fa40abc70 0x7f0fa406d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:50.186 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.183+0000 7f0fa3fff640 1 -- 172.21.15.29:0/1243916239 >> v2:172.21.15.102:3300/0 conn(0x7f0fa40abc70 msgr2=0x7f0fa406d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:45:50.186 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.183+0000 7f0fa3fff640 1 --2- 172.21.15.29:0/1243916239 >> v2:172.21.15.102:3300/0 conn(0x7f0fa40abc70 0x7f0fa406d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:50.186 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.183+0000 7f0fa3fff640 1 -- 172.21.15.29:0/1243916239 >> v1:172.21.15.29:6789/0 conn(0x7f0fa40ac900 legacy=0x7f0fa40acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:50.186 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.183+0000 7f0fa2ffd640 1 -- 172.21.15.29:0/1243916239 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3145883202 0 0) 0x7f0fa406dce0 con 0x7f0fa40ac900 2024-04-25T19:45:50.186 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.183+0000 7f0fa3fff640 1 -- 172.21.15.29:0/1243916239 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fa406d940 con 0x7f0fa41479f0 2024-04-25T19:45:50.186 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.183+0000 7f0fa3fff640 1 --2- 172.21.15.29:0/1243916239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fa41479f0 0x7f0fa4070840 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0fa406cbf0 tx=0x7f0f8c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d9a25c36fe35b9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:45:50.186 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.184+0000 7f0fa2ffd640 1 -- 172.21.15.29:0/1243916239 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c035020 con 0x7f0fa41479f0 2024-04-25T19:45:50.186 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.184+0000 7f0fa2ffd640 1 -- 172.21.15.29:0/1243916239 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f8c031830 con 0x7f0fa41479f0 2024-04-25T19:45:50.186 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.184+0000 7f0fa2ffd640 1 -- 172.21.15.29:0/1243916239 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c031c00 con 0x7f0fa41479f0 2024-04-25T19:45:50.187 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.184+0000 7f0faa8a2640 1 -- 172.21.15.29:0/1243916239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fa41479f0 msgr2=0x7f0fa4070840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:50.187 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.184+0000 7f0faa8a2640 1 --2- 172.21.15.29:0/1243916239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fa41479f0 0x7f0fa4070840 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0fa406cbf0 tx=0x7f0f8c02b8d0 comp rx=0 tx=0).stop 2024-04-25T19:45:50.187 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.184+0000 7f0faa8a2640 1 -- 172.21.15.29:0/1243916239 shutdown_connections 2024-04-25T19:45:50.187 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.184+0000 7f0faa8a2640 1 --2- 172.21.15.29:0/1243916239 >> v2:172.21.15.102:3300/0 conn(0x7f0fa40abc70 0x7f0fa406d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:50.187 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.184+0000 7f0faa8a2640 1 --2- 172.21.15.29:0/1243916239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fa41479f0 0x7f0fa4070840 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:50.187 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.184+0000 7f0faa8a2640 1 -- 172.21.15.29:0/1243916239 >> 172.21.15.29:0/1243916239 conn(0x7f0fa40bc770 msgr2=0x7f0fa40bcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:45:50.187 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.184+0000 7f0faa8a2640 1 -- 172.21.15.29:0/1243916239 shutdown_connections 2024-04-25T19:45:50.187 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.184+0000 7f0faa8a2640 1 -- 172.21.15.29:0/1243916239 wait complete. 2024-04-25T19:45:50.189 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.186+0000 7f0faa8a2640 1 Processor -- start 2024-04-25T19:45:50.190 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.186+0000 7f0faa8a2640 1 -- start start 2024-04-25T19:45:50.190 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.187+0000 7f0faa8a2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fa40abc70 0x7f0fa40a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:50.190 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.187+0000 7f0faa8a2640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0fa40ac900 0x7f0fa41dcc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:50.190 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.187+0000 7f0faa8a2640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fa41479f0 0x7f0fa4144130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:50.190 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.187+0000 7f0faa8a2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa406e9f0 con 0x7f0fa40abc70 2024-04-25T19:45:50.190 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.187+0000 7f0faa8a2640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa406e870 con 0x7f0fa41479f0 2024-04-25T19:45:50.190 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.187+0000 7f0faa8a2640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f0fa40ac080 con 0x7f0fa40ac900 2024-04-25T19:45:50.190 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.187+0000 7f0fa3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fa40abc70 0x7f0fa40a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:50.190 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.187+0000 7f0fa37fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0fa40ac900 0x7f0fa41dcc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:50.190 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.187+0000 7f0fa3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fa40abc70 0x7f0fa40a1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45822/0 (socket says 172.21.15.29:45822) 2024-04-25T19:45:50.190 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.187+0000 7f0fa3fff640 1 -- 172.21.15.29:0/2237717295 learned_addr learned my addr 172.21.15.29:0/2237717295 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:45:50.191 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.187+0000 7f0fa8e3f640 1 --2- 172.21.15.29:0/2237717295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fa41479f0 0x7f0fa4144130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:50.191 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.187+0000 7f0fa3fff640 1 -- 172.21.15.29:0/2237717295 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0fa40ac900 msgr2=0x7f0fa41dcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:50.191 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.187+0000 7f0fa3fff640 1 --2- 172.21.15.29:0/2237717295 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0fa40ac900 0x7f0fa41dcc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:50.191 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.187+0000 7f0fa3fff640 1 -- 172.21.15.29:0/2237717295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fa41479f0 msgr2=0x7f0fa4144130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:50.191 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.187+0000 7f0fa3fff640 1 --2- 172.21.15.29:0/2237717295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fa41479f0 0x7f0fa4144130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:50.191 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.187+0000 7f0fa3fff640 1 -- 172.21.15.29:0/2237717295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fa41ce820 con 0x7f0fa40abc70 2024-04-25T19:45:50.191 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.187+0000 7f0fa37fe640 1 --2- 172.21.15.29:0/2237717295 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0fa40ac900 0x7f0fa41dcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:45:50.191 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.187+0000 7f0fa8e3f640 1 --2- 172.21.15.29:0/2237717295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fa41479f0 0x7f0fa4144130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:45:50.191 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.188+0000 7f0fa3fff640 1 --2- 172.21.15.29:0/2237717295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fa40abc70 0x7f0fa40a1720 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c009890 tx=0x7f0f8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:45:50.191 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.188+0000 7f0fa0ff9640 1 -- 172.21.15.29:0/2237717295 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c040070 con 0x7f0fa40abc70 2024-04-25T19:45:50.191 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.188+0000 7f0fa0ff9640 1 -- 172.21.15.29:0/2237717295 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f8c039940 con 0x7f0fa40abc70 2024-04-25T19:45:50.192 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.188+0000 7f0faa8a2640 1 -- 172.21.15.29:0/2237717295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fa4144df0 con 0x7f0fa40abc70 2024-04-25T19:45:50.192 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.188+0000 7f0fa0ff9640 1 -- 172.21.15.29:0/2237717295 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c035040 con 0x7f0fa40abc70 2024-04-25T19:45:50.192 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.188+0000 7f0faa8a2640 1 -- 172.21.15.29:0/2237717295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fa40a1e40 con 0x7f0fa40abc70 2024-04-25T19:45:50.192 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.189+0000 7f0faa8a2640 1 -- 172.21.15.29:0/2237717295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fa40692d0 con 0x7f0fa40abc70 2024-04-25T19:45:50.193 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.190+0000 7f0fa0ff9640 1 -- 172.21.15.29:0/2237717295 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f0f8c038020 con 0x7f0fa40abc70 2024-04-25T19:45:50.193 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.190+0000 7f0fa0ff9640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:45:50.193 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.191+0000 7f0fa0ff9640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:45:50.193 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.191+0000 7f0fa0ff9640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:45:50.193 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.191+0000 7f0fa0ff9640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:45:50.193 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.191+0000 7f0fa0ff9640 1 --2- 172.21.15.29:0/2237717295 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f0f840ac530 0x7f0f840aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:50.193 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.191+0000 7f0fa0ff9640 1 -- 172.21.15.29:0/2237717295 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 4450+0+0 (secure 0 0 0) 0x7f0f8c0fb420 con 0x7f0fa40abc70 2024-04-25T19:45:50.193 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.191+0000 7f0fa37fe640 1 --2- 172.21.15.29:0/2237717295 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f0f840ac530 0x7f0f840aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:50.193 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.192+0000 7f0fa37fe640 1 --2- 172.21.15.29:0/2237717295 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f0f840ac530 0x7f0f840aea40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0fa40ba0f0 tx=0x7f0f94008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:45:50.194 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.194+0000 7f0fa0ff9640 1 -- 172.21.15.29:0/2237717295 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0f8c0c6010 con 0x7f0fa40abc70 2024-04-25T19:45:50.381 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.381+0000 7f0faa8a2640 1 -- 172.21.15.29:0/2237717295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/localpool/failure_domain}] v 0) -- 0x7f0fa4145a20 con 0x7f0fa40abc70 2024-04-25T19:45:50.386 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.386+0000 7f0fa0ff9640 1 -- 172.21.15.29:0/2237717295 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/localpool/failure_domain}]=0 v7)=0 v7) ==== 128+0+0 (secure 0 0 0) 0x7f0f8c0c5910 con 0x7f0fa40abc70 2024-04-25T19:45:50.498 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.498+0000 7f0faa8a2640 1 -- 172.21.15.29:0/2237717295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/localpool/failure_domain}] v 0) -- 0x7f0fa40ae850 con 0x7f0fa40abc70 2024-04-25T19:45:50.498 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.498+0000 7f0fa0ff9640 1 -- 172.21.15.29:0/2237717295 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mgr/localpool/failure_domain}]=0 v7)=0 v7) ==== 128+0+0 (secure 0 0 0) 0x7f0f8c0c9cc0 con 0x7f0fa40abc70 2024-04-25T19:45:50.501 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.500+0000 7f0faa8a2640 10 mgrc shutdown 2024-04-25T19:45:50.501 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.500+0000 7f0faa8a2640 1 -- 172.21.15.29:0/2237717295 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f0f840ac530 msgr2=0x7f0f840aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:50.501 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.500+0000 7f0faa8a2640 1 --2- 172.21.15.29:0/2237717295 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f0f840ac530 0x7f0f840aea40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0fa40ba0f0 tx=0x7f0f94008040 comp rx=0 tx=0).stop 2024-04-25T19:45:50.501 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.500+0000 7f0faa8a2640 1 -- 172.21.15.29:0/2237717295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fa40abc70 msgr2=0x7f0fa40a1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:50.501 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.500+0000 7f0faa8a2640 1 --2- 172.21.15.29:0/2237717295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fa40abc70 0x7f0fa40a1720 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c009890 tx=0x7f0f8c0046c0 comp rx=0 tx=0).stop 2024-04-25T19:45:50.501 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.500+0000 7f0faa8a2640 1 -- 172.21.15.29:0/2237717295 shutdown_connections 2024-04-25T19:45:50.501 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.500+0000 7f0faa8a2640 1 --2- 172.21.15.29:0/2237717295 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f0f840ac530 0x7f0f840aea40 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:50.501 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.500+0000 7f0faa8a2640 1 --2- 172.21.15.29:0/2237717295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fa41479f0 0x7f0fa4144130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:50.501 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.500+0000 7f0faa8a2640 1 --2- 172.21.15.29:0/2237717295 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0fa40ac900 0x7f0fa41dcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:50.501 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.500+0000 7f0faa8a2640 1 --2- 172.21.15.29:0/2237717295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fa40abc70 0x7f0fa40a1720 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:50.501 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.500+0000 7f0faa8a2640 1 -- 172.21.15.29:0/2237717295 >> 172.21.15.29:0/2237717295 conn(0x7f0fa40bc770 msgr2=0x7f0fa41d32c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:45:50.501 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.500+0000 7f0faa8a2640 1 -- 172.21.15.29:0/2237717295 shutdown_connections 2024-04-25T19:45:50.501 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.500+0000 7f0faa8a2640 1 -- 172.21.15.29:0/2237717295 wait complete. 2024-04-25T19:45:50.510 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph mgr module enable localpool 2024-04-25T19:45:50.603 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.602+0000 7f352b555640 1 Processor -- start 2024-04-25T19:45:50.603 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.603+0000 7f352b555640 1 -- start start 2024-04-25T19:45:50.608 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.604+0000 7f352b555640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f352413d0c0 0x7f35240b6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:50.608 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.604+0000 7f352b555640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f352414b0a0 0x7f35240b6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:50.608 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.604+0000 7f352b555640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f352406f5a0 con 0x7f352413d4d0 2024-04-25T19:45:50.608 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.604+0000 7f352b555640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f352406f860 con 0x7f352414b0a0 2024-04-25T19:45:50.608 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.604+0000 7f352b555640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f352414a470 con 0x7f352413d0c0 2024-04-25T19:45:50.608 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.604+0000 7f35292f1640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f352413d0c0 0x7f35240b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:50.609 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.604+0000 7f35292f1640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f352413d0c0 0x7f35240b6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:56002/0 (socket says 172.21.15.29:56002) 2024-04-25T19:45:50.609 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.604+0000 7f35292f1640 1 -- 172.21.15.29:0/2398498012 learned_addr learned my addr 172.21.15.29:0/2398498012 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:45:50.609 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.604+0000 7f35292f1640 1 -- 172.21.15.29:0/2398498012 >> v2:172.21.15.102:3300/0 conn(0x7f352414b0a0 msgr2=0x7f35240b6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:45:50.609 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.604+0000 7f35292f1640 1 --2- 172.21.15.29:0/2398498012 >> v2:172.21.15.102:3300/0 conn(0x7f352414b0a0 0x7f35240b6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:50.609 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.604+0000 7f35292f1640 1 -- 172.21.15.29:0/2398498012 >> v1:172.21.15.102:6789/0 conn(0x7f352413d4d0 legacy=0x7f352414a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:45:50.609 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.604+0000 7f35292f1640 1 -- 172.21.15.29:0/2398498012 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35240afb80 con 0x7f352413d0c0 2024-04-25T19:45:50.610 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.605+0000 7f35292f1640 1 --2- 172.21.15.29:0/2398498012 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f352413d0c0 0x7f35240b6200 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f351c009e50 tx=0x7f351c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb3588cf827f8e4b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:45:50.610 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.605+0000 7f3523fff640 1 -- 172.21.15.29:0/2398498012 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351c02cb00 con 0x7f352413d0c0 2024-04-25T19:45:50.610 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.605+0000 7f3523fff640 1 -- 172.21.15.29:0/2398498012 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f351c02cca0 con 0x7f352413d0c0 2024-04-25T19:45:50.610 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.605+0000 7f3523fff640 1 -- 172.21.15.29:0/2398498012 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351c034610 con 0x7f352413d0c0 2024-04-25T19:45:50.610 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.606+0000 7f352b555640 1 -- 172.21.15.29:0/2398498012 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f352413d0c0 msgr2=0x7f35240b6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:50.610 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.606+0000 7f352b555640 1 --2- 172.21.15.29:0/2398498012 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f352413d0c0 0x7f35240b6200 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f351c009e50 tx=0x7f351c02bf60 comp rx=0 tx=0).stop 2024-04-25T19:45:50.611 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.606+0000 7f352b555640 1 -- 172.21.15.29:0/2398498012 shutdown_connections 2024-04-25T19:45:50.611 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.606+0000 7f352b555640 1 --2- 172.21.15.29:0/2398498012 >> v2:172.21.15.102:3300/0 conn(0x7f352414b0a0 0x7f35240b6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:50.611 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.606+0000 7f352b555640 1 --2- 172.21.15.29:0/2398498012 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f352413d0c0 0x7f35240b6200 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:50.611 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.606+0000 7f352b555640 1 -- 172.21.15.29:0/2398498012 >> 172.21.15.29:0/2398498012 conn(0x7f3524073250 msgr2=0x7f3524073660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:45:50.611 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.606+0000 7f352b555640 1 -- 172.21.15.29:0/2398498012 shutdown_connections 2024-04-25T19:45:50.611 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.606+0000 7f352b555640 1 -- 172.21.15.29:0/2398498012 wait complete. 2024-04-25T19:45:50.611 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.607+0000 7f352b555640 1 Processor -- start 2024-04-25T19:45:50.612 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.607+0000 7f352b555640 1 -- start start 2024-04-25T19:45:50.616 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.608+0000 7f352b555640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f352413d0c0 0x7f352414e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:50.616 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.608+0000 7f352b555640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f352413d4d0 0x7f35241e04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:50.616 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.608+0000 7f352b555640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f352414b0a0 0x7f35241e09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:50.616 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.608+0000 7f352b555640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3524151960 con 0x7f352413d4d0 2024-04-25T19:45:50.616 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.608+0000 7f352b555640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f35241517e0 con 0x7f352413d0c0 2024-04-25T19:45:50.616 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.608+0000 7f352b555640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f3524151ae0 con 0x7f352414b0a0 2024-04-25T19:45:50.616 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.608+0000 7f3528af0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f352413d4d0 0x7f35241e04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:50.616 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.608+0000 7f3529af2640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f352414b0a0 0x7f35241e09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:50.616 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.608+0000 7f3528af0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f352413d4d0 0x7f35241e04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45828/0 (socket says 172.21.15.29:45828) 2024-04-25T19:45:50.616 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.608+0000 7f3529af2640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f352414b0a0 0x7f35241e09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:56014/0 (socket says 172.21.15.29:56014) 2024-04-25T19:45:50.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.608+0000 7f3528af0640 1 -- 172.21.15.29:0/1257036963 learned_addr learned my addr 172.21.15.29:0/1257036963 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:45:50.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.608+0000 7f35292f1640 1 --2- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f352413d0c0 0x7f352414e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:50.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.609+0000 7f3528af0640 1 -- 172.21.15.29:0/1257036963 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f352414b0a0 msgr2=0x7f35241e09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:50.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.609+0000 7f3528af0640 1 --2- 172.21.15.29:0/1257036963 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f352414b0a0 0x7f35241e09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:50.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.609+0000 7f3528af0640 1 -- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f352413d0c0 msgr2=0x7f352414e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:50.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.609+0000 7f3528af0640 1 --2- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f352413d0c0 0x7f352414e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:50.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.609+0000 7f3528af0640 1 -- 172.21.15.29:0/1257036963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3524131c90 con 0x7f352413d4d0 2024-04-25T19:45:50.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.609+0000 7f3529af2640 1 --2- 172.21.15.29:0/1257036963 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f352414b0a0 0x7f35241e09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:45:50.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.609+0000 7f3528af0640 1 --2- 172.21.15.29:0/1257036963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f352413d4d0 0x7f35241e04a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f351800b3e0 tx=0x7f351800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:45:50.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.609+0000 7f3521ffb640 1 -- 172.21.15.29:0/1257036963 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3518015020 con 0x7f352413d4d0 2024-04-25T19:45:50.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.609+0000 7f3521ffb640 1 -- 172.21.15.29:0/1257036963 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3518011850 con 0x7f352413d4d0 2024-04-25T19:45:50.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.609+0000 7f352b555640 1 -- 172.21.15.29:0/1257036963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3524138670 con 0x7f352413d4d0 2024-04-25T19:45:50.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.609+0000 7f3521ffb640 1 -- 172.21.15.29:0/1257036963 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3518011c40 con 0x7f352413d4d0 2024-04-25T19:45:50.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.609+0000 7f352b555640 1 -- 172.21.15.29:0/1257036963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f352414e0c0 con 0x7f352413d4d0 2024-04-25T19:45:50.618 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.610+0000 7f352b555640 1 -- 172.21.15.29:0/1257036963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35240692d0 con 0x7f352413d4d0 2024-04-25T19:45:50.618 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.612+0000 7f3521ffb640 1 -- 172.21.15.29:0/1257036963 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 144859+0+0 (secure 0 0 0) 0x7f3518018020 con 0x7f352413d4d0 2024-04-25T19:45:50.618 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.612+0000 7f3521ffb640 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-25T19:45:50.618 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.612+0000 7f3521ffb640 4 mgrc handle_mgr_map Got map version 5 2024-04-25T19:45:50.618 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.612+0000 7f3521ffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:45:50.618 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.612+0000 7f3521ffb640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:45:50.618 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.612+0000 7f3521ffb640 1 --2- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f35100ac530 0x7f35100aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:50.618 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.612+0000 7f3521ffb640 1 -- 172.21.15.29:0/1257036963 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 4450+0+0 (secure 0 0 0) 0x7f35180d9800 con 0x7f352413d4d0 2024-04-25T19:45:50.618 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.613+0000 7f35292f1640 1 --2- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f35100ac530 0x7f35100aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:50.618 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.613+0000 7f35292f1640 1 --2- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f35100ac530 0x7f35100aea40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f351c035000 tx=0x7f351c030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:45:50.619 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.616+0000 7f3521ffb640 1 -- 172.21.15.29:0/1257036963 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f351801f020 con 0x7f352413d4d0 2024-04-25T19:45:50.851 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:50.850+0000 7f352b555640 1 -- 172.21.15.29:0/1257036963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "localpool"} v 0) -- 0x7f352406cff0 con 0x7f352413d4d0 2024-04-25T19:45:51.386 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:51.386+0000 7f3521ffb640 1 -- 172.21.15.29:0/1257036963 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "localpool"}]=0 v6) ==== 88+0+0 (secure 0 0 0) 0x7f35180a43f0 con 0x7f352413d4d0 2024-04-25T19:45:51.392 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:51.391+0000 7f3521ffb640 1 -- 172.21.15.29:0/1257036963 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 6) ==== 144872+0+0 (secure 0 0 0) 0x7f351809fef0 con 0x7f352413d4d0 2024-04-25T19:45:51.392 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:51.391+0000 7f3521ffb640 20 mgrc handle_mgr_map mgrmap(e 6) 2024-04-25T19:45:51.392 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:51.391+0000 7f3521ffb640 4 mgrc handle_mgr_map Got map version 6 2024-04-25T19:45:51.392 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:51.391+0000 7f3521ffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:45:51.416 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:51.415+0000 7f35292f1640 1 -- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f35100ac530 msgr2=0x7f35100aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-25T19:45:51.416 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:51.415+0000 7f35292f1640 1 -- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f35100ac530 msgr2=0x7f35100aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T19:45:51.416 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:51.415+0000 7f35292f1640 1 --2- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f35100ac530 0x7f35100aea40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f351c035000 tx=0x7f351c030000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T19:45:51.416 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:51.415+0000 7f35292f1640 1 --2- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f35100ac530 0x7f35100aea40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f351c035000 tx=0x7f351c030000 comp rx=0 tx=0).stop 2024-04-25T19:45:51.416 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:51.415+0000 7f3521ffb640 4 mgrc ms_handle_reset ms_handle_reset con 0x7f35100ac530 2024-04-25T19:45:51.417 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:51.415+0000 7f3521ffb640 4 mgrc reconnect Terminating session with v2:172.21.15.102:6824/1439704316 2024-04-25T19:45:51.417 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:51.415+0000 7f3521ffb640 1 -- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f35100ac530 msgr2=0x7f35100aea40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T19:45:51.417 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:51.415+0000 7f3521ffb640 1 --2- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f35100ac530 0x7f35100aea40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:51.417 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:51.415+0000 7f3521ffb640 4 mgrc reconnect waiting to retry connect until 586.712264s 2024-04-25T19:45:51.540 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:51.540+0000 7f8ac6c32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T19:45:51.548 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:51.547+0000 7f19f504d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T19:45:51.548 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:51.547+0000 7f3d2a353040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T19:45:51.550 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:51.551+0000 7f352b555640 1 -- 172.21.15.29:0/1257036963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "localpool"} v 0) -- 0x7f34fc0050a0 con 0x7f352413d4d0 2024-04-25T19:45:51.613 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:51.612+0000 7f350bfff640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:45:51.614 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:51.612+0000 7f350bfff640 1 --2- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f35100ac530 0x7f35241e0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:51.614 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:51.613+0000 7f35292f1640 1 -- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f35100ac530 msgr2=0x7f35241e0f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6824/1439704316 2024-04-25T19:45:51.614 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:51.613+0000 7f35292f1640 1 --2- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f35100ac530 0x7f35241e0f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T19:45:51.659 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:51.659+0000 7f8ac6c32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:45:51.662 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:51.662+0000 7f3d2a353040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:45:51.663 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:51.662+0000 7f19f504d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:45:51.708 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:51.708+0000 7f8ac6c32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T19:45:51.710 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:51.710+0000 7f19f504d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T19:45:51.711 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:51.710+0000 7f3d2a353040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T19:45:51.814 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:51.813+0000 7f35292f1640 1 -- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f35100ac530 msgr2=0x7f35241e0f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6824/1439704316 2024-04-25T19:45:51.814 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:51.813+0000 7f35292f1640 1 --2- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f35100ac530 0x7f35241e0f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T19:45:51.918 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:51.917+0000 7f19f504d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T19:45:51.918 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:51.917+0000 7f3d2a353040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T19:45:51.923 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:51.922+0000 7f8ac6c32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T19:45:51.962 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:51.961+0000 7f19f504d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T19:45:51.962 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:51.961+0000 7f3d2a353040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T19:45:51.968 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:51.967+0000 7f8ac6c32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T19:45:52.215 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.214+0000 7f35292f1640 1 -- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f35100ac530 msgr2=0x7f35241e0f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6824/1439704316 2024-04-25T19:45:52.215 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.214+0000 7f35292f1640 1 --2- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f35100ac530 0x7f35241e0f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T19:45:52.389 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.388+0000 7f3521ffb640 1 -- 172.21.15.29:0/1257036963 <== mon.0 v2:172.21.15.29:3300/0 9 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "localpool"}]=0 module 'localpool' is already enabled v7) ==== 125+0+0 (secure 0 0 0) 0x7f351809bed0 con 0x7f352413d4d0 2024-04-25T19:45:52.389 INFO:tasks.workunit.client.0.smithi029.stderr:module 'localpool' is already enabled 2024-04-25T19:45:52.391 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.390+0000 7f352b555640 10 mgrc shutdown 2024-04-25T19:45:52.391 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.390+0000 7f352b555640 1 -- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f35100ac530 msgr2=0x7f35241e0f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T19:45:52.391 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.390+0000 7f352b555640 1 --2- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f35100ac530 0x7f35241e0f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:52.391 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.390+0000 7f352b555640 1 -- 172.21.15.29:0/1257036963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f352413d4d0 msgr2=0x7f35241e04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:52.393 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.391+0000 7f3521ffb640 1 -- 172.21.15.29:0/1257036963 <== mon.0 v2:172.21.15.29:3300/0 10 ==== mgrmap(e 7) ==== 144872+0+0 (secure 0 0 0) 0x7f3518011c40 con 0x7f352413d4d0 2024-04-25T19:45:52.393 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.391+0000 7f352b555640 1 --2- 172.21.15.29:0/1257036963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f352413d4d0 0x7f35241e04a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f351800b3e0 tx=0x7f351800b8a0 comp rx=0 tx=0).stop 2024-04-25T19:45:52.393 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.391+0000 7f3521ffb640 20 mgrc handle_mgr_map mgrmap(e 7) 2024-04-25T19:45:52.393 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.391+0000 7f352b555640 1 -- 172.21.15.29:0/1257036963 shutdown_connections 2024-04-25T19:45:52.393 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.391+0000 7f352b555640 1 --2- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f35100ac530 0x7f35241e0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:52.393 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.391+0000 7f352b555640 1 --2- 172.21.15.29:0/1257036963 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f352414b0a0 0x7f35241e09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:52.393 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.391+0000 7f352b555640 1 --2- 172.21.15.29:0/1257036963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f352413d4d0 0x7f35241e04a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:52.393 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.391+0000 7f352b555640 1 --2- 172.21.15.29:0/1257036963 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f352413d0c0 0x7f352414e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:52.393 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.391+0000 7f352b555640 1 -- 172.21.15.29:0/1257036963 >> 172.21.15.29:0/1257036963 conn(0x7f3524073250 msgr2=0x7f35240b85e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:45:52.393 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.392+0000 7f3521ffb640 4 mgrc handle_mgr_map Got map version 7 2024-04-25T19:45:52.394 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.392+0000 7f3521ffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:45:52.394 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.392+0000 7f3521ffb640 4 mgrc reconnect waiting to retry connect until 587.712390s 2024-04-25T19:45:52.394 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.392+0000 7f352b555640 1 -- 172.21.15.29:0/1257036963 shutdown_connections 2024-04-25T19:45:52.394 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.392+0000 7f352b555640 1 -- 172.21.15.29:0/1257036963 wait complete. 2024-04-25T19:45:52.405 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph osd pool ls 2024-04-25T19:45:52.405 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep '^by-host-' 2024-04-25T19:45:52.475 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:52.474+0000 7f3d2a353040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T19:45:52.475 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:52.475+0000 7f19f504d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T19:45:52.488 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:52.487+0000 7f8ac6c32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T19:45:52.496 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.495+0000 7f37ab3ab640 1 Processor -- start 2024-04-25T19:45:52.496 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.495+0000 7f37ab3ab640 1 -- start start 2024-04-25T19:45:52.499 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.496+0000 7f37ab3ab640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f37a414e750 0x7f37a4150b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:52.499 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.496+0000 7f37ab3ab640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37a40abdc0 con 0x7f37a40746c0 2024-04-25T19:45:52.500 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.496+0000 7f37ab3ab640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37a406c5c0 con 0x7f37a4144330 2024-04-25T19:45:52.500 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.496+0000 7f37ab3ab640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f37a4074ad0 con 0x7f37a414e750 2024-04-25T19:45:52.500 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.497+0000 7f37a9147640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f37a40746c0 0x7f37a40ab260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50880/0 (socket says 172.21.15.29:50880) 2024-04-25T19:45:52.500 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.497+0000 7f37a9147640 1 -- 172.21.15.29:0/4283649001 learned_addr learned my addr 172.21.15.29:0/4283649001 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:45:52.500 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.497+0000 7f37a9948640 1 --2- 172.21.15.29:0/4283649001 >> v2:172.21.15.102:3300/0 conn(0x7f37a414e750 0x7f37a4150b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:52.500 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.497+0000 7f379bfff640 1 -- 172.21.15.29:0/4283649001 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1823202158 0 0) 0x7f37a40abdc0 con 0x7f37a40746c0 2024-04-25T19:45:52.500 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.497+0000 7f379bfff640 1 -- 172.21.15.29:0/4283649001 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3790003520 con 0x7f37a40746c0 2024-04-25T19:45:52.500 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.497+0000 7f379bfff640 1 -- 172.21.15.29:0/4283649001 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3224740769 0 0) 0x7f37a406c5c0 con 0x7f37a4144330 2024-04-25T19:45:52.500 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.497+0000 7f379bfff640 1 -- 172.21.15.29:0/4283649001 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37a40abdc0 con 0x7f37a4144330 2024-04-25T19:45:52.500 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.497+0000 7f37a9948640 1 -- 172.21.15.29:0/4283649001 >> v1:172.21.15.102:6789/0 conn(0x7f37a4144330 legacy=0x7f37a4144740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:52.500 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.497+0000 7f37a9948640 1 -- 172.21.15.29:0/4283649001 >> v1:172.21.15.29:6789/0 conn(0x7f37a40746c0 legacy=0x7f37a40ab260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:52.500 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.497+0000 7f37a9948640 1 -- 172.21.15.29:0/4283649001 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37a40ab970 con 0x7f37a414e750 2024-04-25T19:45:52.501 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.498+0000 7f37a9948640 1 --2- 172.21.15.29:0/4283649001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37a414e750 0x7f37a4150b70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f379c009d90 tx=0x7f379c02bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dacb777b5cf82d1e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:45:52.501 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.498+0000 7f379bfff640 1 -- 172.21.15.29:0/4283649001 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f379c039070 con 0x7f37a414e750 2024-04-25T19:45:52.501 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.498+0000 7f379bfff640 1 -- 172.21.15.29:0/4283649001 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f379c02cd80 con 0x7f37a414e750 2024-04-25T19:45:52.501 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.498+0000 7f379bfff640 1 -- 172.21.15.29:0/4283649001 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f379c034660 con 0x7f37a414e750 2024-04-25T19:45:52.504 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.499+0000 7f37ab3ab640 1 -- 172.21.15.29:0/4283649001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37a414e750 msgr2=0x7f37a4150b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:52.504 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.499+0000 7f37ab3ab640 1 --2- 172.21.15.29:0/4283649001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37a414e750 0x7f37a4150b70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f379c009d90 tx=0x7f379c02bfd0 comp rx=0 tx=0).stop 2024-04-25T19:45:52.504 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.499+0000 7f37ab3ab640 1 -- 172.21.15.29:0/4283649001 shutdown_connections 2024-04-25T19:45:52.504 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.499+0000 7f37ab3ab640 1 --2- 172.21.15.29:0/4283649001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37a414e750 0x7f37a4150b70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:52.504 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.499+0000 7f37ab3ab640 1 -- 172.21.15.29:0/4283649001 >> 172.21.15.29:0/4283649001 conn(0x7f37a40a8940 msgr2=0x7f37a40a8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:45:52.504 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.499+0000 7f37ab3ab640 1 -- 172.21.15.29:0/4283649001 shutdown_connections 2024-04-25T19:45:52.504 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.499+0000 7f37ab3ab640 1 -- 172.21.15.29:0/4283649001 wait complete. 2024-04-25T19:45:52.504 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.500+0000 7f37ab3ab640 1 Processor -- start 2024-04-25T19:45:52.504 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.500+0000 7f37ab3ab640 1 -- start start 2024-04-25T19:45:52.505 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.501+0000 7f37ab3ab640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f37a40746c0 0x7f37a4147550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:52.505 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.501+0000 7f37ab3ab640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37a4144330 0x7f37a413ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:52.505 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.501+0000 7f37ab3ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37a414e750 0x7f37a40a3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:52.505 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.501+0000 7f37ab3ab640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f37a40aad90 con 0x7f37a414e750 2024-04-25T19:45:52.505 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.501+0000 7f37ab3ab640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f37a40aac10 con 0x7f37a4144330 2024-04-25T19:45:52.505 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.501+0000 7f37ab3ab640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f37a41510b0 con 0x7f37a40746c0 2024-04-25T19:45:52.505 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.501+0000 7f37a9948640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37a414e750 0x7f37a40a3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:52.505 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.501+0000 7f37a9147640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f37a40746c0 0x7f37a4147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:52.505 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.501+0000 7f37a9948640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37a414e750 0x7f37a40a3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45844/0 (socket says 172.21.15.29:45844) 2024-04-25T19:45:52.505 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.501+0000 7f37a9948640 1 -- 172.21.15.29:0/94555016 learned_addr learned my addr 172.21.15.29:0/94555016 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:45:52.505 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.502+0000 7f37a8946640 1 --2- 172.21.15.29:0/94555016 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37a4144330 0x7f37a413ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:52.505 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.502+0000 7f37a9948640 1 -- 172.21.15.29:0/94555016 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f37a40746c0 msgr2=0x7f37a4147550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:52.506 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.502+0000 7f37a9948640 1 --2- 172.21.15.29:0/94555016 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f37a40746c0 0x7f37a4147550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:52.506 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.502+0000 7f37a9948640 1 -- 172.21.15.29:0/94555016 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37a4144330 msgr2=0x7f37a413ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:52.506 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.502+0000 7f37a9948640 1 --2- 172.21.15.29:0/94555016 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37a4144330 0x7f37a413ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:52.506 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.502+0000 7f37a9948640 1 -- 172.21.15.29:0/94555016 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37a40a5370 con 0x7f37a414e750 2024-04-25T19:45:52.506 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.502+0000 7f37a9948640 1 --2- 172.21.15.29:0/94555016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37a414e750 0x7f37a40a3fa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f379c009d90 tx=0x7f379c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:45:52.506 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.502+0000 7f3799ffb640 1 -- 172.21.15.29:0/94555016 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f379c039050 con 0x7f37a414e750 2024-04-25T19:45:52.506 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.502+0000 7f3799ffb640 1 -- 172.21.15.29:0/94555016 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f379c045070 con 0x7f37a414e750 2024-04-25T19:45:52.506 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.502+0000 7f37ab3ab640 1 -- 172.21.15.29:0/94555016 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37a4147e20 con 0x7f37a414e750 2024-04-25T19:45:52.506 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.502+0000 7f3799ffb640 1 -- 172.21.15.29:0/94555016 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f379c009450 con 0x7f37a414e750 2024-04-25T19:45:52.506 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.503+0000 7f37ab3ab640 1 -- 172.21.15.29:0/94555016 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37a4150a60 con 0x7f37a414e750 2024-04-25T19:45:52.507 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.504+0000 7f37ab3ab640 1 -- 172.21.15.29:0/94555016 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37a40692d0 con 0x7f37a414e750 2024-04-25T19:45:52.508 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.505+0000 7f3799ffb640 1 -- 172.21.15.29:0/94555016 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 7) ==== 144872+0+0 (secure 0 0 0) 0x7f379c03e850 con 0x7f37a414e750 2024-04-25T19:45:52.508 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.505+0000 7f3799ffb640 20 mgrc handle_mgr_map mgrmap(e 7) 2024-04-25T19:45:52.508 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.505+0000 7f3799ffb640 4 mgrc handle_mgr_map Got map version 7 2024-04-25T19:45:52.508 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.505+0000 7f3799ffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:45:52.508 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.505+0000 7f3799ffb640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] 2024-04-25T19:45:52.508 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.505+0000 7f3799ffb640 1 --2- 172.21.15.29:0/94555016 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f37880ac630 0x7f37880aeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:52.508 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.506+0000 7f37a9147640 1 -- 172.21.15.29:0/94555016 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f37880ac630 msgr2=0x7f37880aeb40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6824/1439704316 2024-04-25T19:45:52.508 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.506+0000 7f3799ffb640 1 -- 172.21.15.29:0/94555016 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 4450+0+0 (secure 0 0 0) 0x7f379c0fb9f0 con 0x7f37a414e750 2024-04-25T19:45:52.508 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.506+0000 7f37a9147640 1 --2- 172.21.15.29:0/94555016 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f37880ac630 0x7f37880aeb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T19:45:52.511 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.510+0000 7f3799ffb640 1 -- 172.21.15.29:0/94555016 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f379c04e020 con 0x7f37a414e750 2024-04-25T19:45:52.563 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:52.562+0000 7f3d2a353040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T19:45:52.563 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:52.562+0000 7f19f504d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T19:45:52.580 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:52.579+0000 7f8ac6c32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T19:45:52.606 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:52.605+0000 7f3d2a353040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T19:45:52.606 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:52.605+0000 7f19f504d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T19:45:52.624 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:52.624+0000 7f8ac6c32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T19:45:52.695 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.694+0000 7f37ab3ab640 1 -- 172.21.15.29:0/94555016 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool ls"} v 0) -- 0x7f37a40744a0 con 0x7f37a414e750 2024-04-25T19:45:52.695 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.694+0000 7f3799ffb640 1 -- 172.21.15.29:0/94555016 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool ls"}]=0 v22) ==== 59+0+37 (secure 0 0 0) 0x7f379c0c6540 con 0x7f37a414e750 2024-04-25T19:45:52.708 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.707+0000 7f37a9147640 1 -- 172.21.15.29:0/94555016 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f37880ac630 msgr2=0x7f37880aeb40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6824/1439704316 2024-04-25T19:45:52.708 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.707+0000 7f37a9147640 1 --2- 172.21.15.29:0/94555016 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f37880ac630 0x7f37880aeb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T19:45:52.773 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:52.772+0000 7f3d2a353040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T19:45:52.773 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:52.772+0000 7f19f504d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T19:45:52.797 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:52.796+0000 7f8ac6c32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T19:45:52.803 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.802+0000 7f37ab3ab640 1 -- 172.21.15.29:0/94555016 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool ls"} v 0) -- 0x7f37a40a4ca0 con 0x7f37a414e750 2024-04-25T19:45:52.803 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.803+0000 7f3799ffb640 1 -- 172.21.15.29:0/94555016 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool ls"}]=0 v22) ==== 59+0+37 (secure 0 0 0) 0x7f379c0c5e40 con 0x7f37a414e750 2024-04-25T19:45:52.806 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.804+0000 7f37ab3ab640 10 mgrc shutdown 2024-04-25T19:45:52.806 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.805+0000 7f37ab3ab640 1 -- 172.21.15.29:0/94555016 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f37880ac630 msgr2=0x7f37880aeb40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T19:45:52.807 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.805+0000 7f37ab3ab640 1 --2- 172.21.15.29:0/94555016 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f37880ac630 0x7f37880aeb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:52.807 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.805+0000 7f37ab3ab640 1 -- 172.21.15.29:0/94555016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37a414e750 msgr2=0x7f37a40a3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:52.807 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.805+0000 7f37ab3ab640 1 --2- 172.21.15.29:0/94555016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37a414e750 0x7f37a40a3fa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f379c009d90 tx=0x7f379c004a40 comp rx=0 tx=0).stop 2024-04-25T19:45:52.807 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.805+0000 7f37ab3ab640 1 -- 172.21.15.29:0/94555016 shutdown_connections 2024-04-25T19:45:52.807 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.805+0000 7f37ab3ab640 1 --2- 172.21.15.29:0/94555016 >> [v2:172.21.15.102:6824/1439704316,v1:172.21.15.102:6825/1439704316] conn(0x7f37880ac630 0x7f37880aeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:52.807 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.805+0000 7f37ab3ab640 1 --2- 172.21.15.29:0/94555016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37a414e750 0x7f37a40a3fa0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:52.807 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.805+0000 7f37ab3ab640 1 --2- 172.21.15.29:0/94555016 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37a4144330 0x7f37a413ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:52.807 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.805+0000 7f37ab3ab640 1 --2- 172.21.15.29:0/94555016 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f37a40746c0 0x7f37a4147550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:52.807 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.805+0000 7f37ab3ab640 1 -- 172.21.15.29:0/94555016 >> 172.21.15.29:0/94555016 conn(0x7f37a40a8940 msgr2=0x7f37a40b81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:45:52.807 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.805+0000 7f37ab3ab640 1 -- 172.21.15.29:0/94555016 shutdown_connections 2024-04-25T19:45:52.807 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:52.805+0000 7f37ab3ab640 1 -- 172.21.15.29:0/94555016 wait complete. 2024-04-25T19:45:52.815 INFO:tasks.workunit.client.0.smithi029.stderr:+ sleep 5 2024-04-25T19:45:52.818 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:52.817+0000 7f3d2a353040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T19:45:52.818 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:52.817+0000 7f19f504d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T19:45:52.845 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:52.844+0000 7f8ac6c32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T19:45:52.868 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:52.867+0000 7f19f504d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T19:45:52.868 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:52.867+0000 7f3d2a353040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T19:45:52.897 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:52.896+0000 7f8ac6c32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T19:45:52.980 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:52.980+0000 7f19f504d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:45:52.981 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:52.980+0000 7f3d2a353040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:45:53.013 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:53.012+0000 7f8ac6c32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:45:53.318 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:53.318+0000 7f3d2a353040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T19:45:53.319 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:53.318+0000 7f19f504d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T19:45:53.352 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:53.351+0000 7f8ac6c32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T19:45:53.538 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:53.538+0000 7f3d2a353040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T19:45:53.539 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:53.538+0000 7f19f504d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T19:45:53.574 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:53.573+0000 7f8ac6c32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T19:45:53.590 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:53.589+0000 7f3d2a353040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T19:45:53.590 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:53.589+0000 7f19f504d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T19:45:53.626 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:53.625+0000 7f8ac6c32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T19:45:53.709 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:53.708+0000 7f3d2a353040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T19:45:53.709 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:53.708+0000 7f19f504d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T19:45:53.745 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:53.745+0000 7f8ac6c32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T19:45:53.986 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:53.986+0000 7f19f504d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T19:45:53.987 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:53.986+0000 7f3d2a353040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T19:45:54.028 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:54.027+0000 7f8ac6c32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T19:45:54.032 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:54.032+0000 7f3d2a353040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T19:45:54.032 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:54.032+0000 7f19f504d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T19:45:54.074 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:54.073+0000 7f8ac6c32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T19:45:54.184 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:54.183+0000 7f3d2a353040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T19:45:54.184 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:54.184+0000 7f19f504d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T19:45:54.225 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:54.224+0000 7f8ac6c32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T19:45:54.321 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:54.320+0000 7f3d2a353040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T19:45:54.321 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:54.320+0000 7f19f504d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T19:45:54.365 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:54.364+0000 7f8ac6c32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T19:45:54.371 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:54.371+0000 7f3d2a353040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T19:45:54.371 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:54.371+0000 7f19f504d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T19:45:54.416 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:54.415+0000 7f8ac6c32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T19:45:54.427 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:54.427+0000 7f3d2a353040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T19:45:54.427 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:54.427+0000 7f19f504d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T19:45:54.472 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:54.471+0000 7f19f504d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T19:45:54.472 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:54.471+0000 7f3d2a353040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T19:45:54.473 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:54.471+0000 7f8ac6c32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T19:45:54.517 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:54.516+0000 7f8ac6c32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T19:45:54.879 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:54.878+0000 7f3d2a353040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T19:45:54.879 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:54.878+0000 7f19f504d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T19:45:54.925 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:54.924+0000 7f8ac6c32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T19:45:54.999 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:54.998+0000 7f3d2a353040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T19:45:54.999 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:54.999+0000 7f19f504d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T19:45:55.052 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:55.051+0000 7f8ac6c32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T19:45:55.067 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:55.066+0000 7f1975318640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:45:55.067 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:55.066+0000 7f1975318640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:45:55.067 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:55.066+0000 7f1975318640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:45:55.067 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:55.066+0000 7f1975318640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:45:55.067 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:55.066+0000 7f1975318640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:45:55.067 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:55.066+0000 7f1978b1f640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:45:55.068 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:55.066+0000 7f1978b1f640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:45:55.068 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:55.066+0000 7f1978b1f640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:45:55.068 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:55.066+0000 7f1978b1f640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:45:55.068 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:55.066+0000 7f1978b1f640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:45:57.817 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph osd pool ls 2024-04-25T19:45:57.817 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep '^by-host-' 2024-04-25T19:45:57.920 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.919+0000 7f81d8bac640 1 Processor -- start 2024-04-25T19:45:57.920 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.919+0000 7f81d8bac640 1 -- start start 2024-04-25T19:45:57.922 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.920+0000 7f81d8bac640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f81d40abeb0 0x7f81d4070840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:57.922 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.920+0000 7f81d8bac640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81d40ac2c0 con 0x7f81d414a990 2024-04-25T19:45:57.922 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.920+0000 7f81d8bac640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81d414aee0 con 0x7f81d414a580 2024-04-25T19:45:57.922 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.920+0000 7f81d8bac640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f81d4070d80 con 0x7f81d40abeb0 2024-04-25T19:45:57.922 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.920+0000 7f81d259c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f81d40abeb0 0x7f81d4070840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:57.922 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.920+0000 7f81d2d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f81d414a990 0x7f81d41518f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43206/0 (socket says 172.21.15.29:43206) 2024-04-25T19:45:57.922 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.920+0000 7f81d259c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f81d40abeb0 0x7f81d4070840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:49812/0 (socket says 172.21.15.29:49812) 2024-04-25T19:45:57.922 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.920+0000 7f81d2d9d640 1 -- 172.21.15.29:0/2653717748 learned_addr learned my addr 172.21.15.29:0/2653717748 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:45:57.923 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.921+0000 7f81d259c640 1 -- 172.21.15.29:0/2653717748 >> v1:172.21.15.102:6789/0 conn(0x7f81d414a580 legacy=0x7f81d406d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:45:57.923 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.921+0000 7f81d1d9b640 1 -- 172.21.15.29:0/2653717748 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 895084793 0 0) 0x7f81cc001490 con 0x7f81d414a990 2024-04-25T19:45:57.923 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.921+0000 7f81d259c640 1 -- 172.21.15.29:0/2653717748 >> v1:172.21.15.29:6789/0 conn(0x7f81d414a990 legacy=0x7f81d41518f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:57.923 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.921+0000 7f81d259c640 1 -- 172.21.15.29:0/2653717748 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d414b180 con 0x7f81d40abeb0 2024-04-25T19:45:57.924 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.921+0000 7f81d259c640 1 --2- 172.21.15.29:0/2653717748 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f81d40abeb0 0x7f81d4070840 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f81c40097d0 tx=0x7f81c402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ef79ba5483961d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:45:57.924 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.922+0000 7f81d1d9b640 1 -- 172.21.15.29:0/2653717748 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c4035020 con 0x7f81d40abeb0 2024-04-25T19:45:57.924 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.922+0000 7f81d1d9b640 1 -- 172.21.15.29:0/2653717748 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81c4031800 con 0x7f81d40abeb0 2024-04-25T19:45:57.924 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.922+0000 7f81d1d9b640 1 -- 172.21.15.29:0/2653717748 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c4031bd0 con 0x7f81d40abeb0 2024-04-25T19:45:57.924 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.922+0000 7f81d8bac640 1 -- 172.21.15.29:0/2653717748 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f81d40abeb0 msgr2=0x7f81d4070840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:57.924 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.922+0000 7f81d8bac640 1 --2- 172.21.15.29:0/2653717748 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f81d40abeb0 0x7f81d4070840 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f81c40097d0 tx=0x7f81c402b8a0 comp rx=0 tx=0).stop 2024-04-25T19:45:57.924 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.922+0000 7f81d8bac640 1 -- 172.21.15.29:0/2653717748 shutdown_connections 2024-04-25T19:45:57.924 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.922+0000 7f81d8bac640 1 --2- 172.21.15.29:0/2653717748 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f81d40abeb0 0x7f81d4070840 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:57.924 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.922+0000 7f81d8bac640 1 -- 172.21.15.29:0/2653717748 >> 172.21.15.29:0/2653717748 conn(0x7f81d40bc870 msgr2=0x7f81d40bcc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:45:57.925 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.922+0000 7f81d8bac640 1 -- 172.21.15.29:0/2653717748 shutdown_connections 2024-04-25T19:45:57.925 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.922+0000 7f81d8bac640 1 -- 172.21.15.29:0/2653717748 wait complete. 2024-04-25T19:45:57.925 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.924+0000 7f81d8bac640 1 Processor -- start 2024-04-25T19:45:57.925 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.924+0000 7f81d8bac640 1 -- start start 2024-04-25T19:45:57.928 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.925+0000 7f81d8bac640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81d40abeb0 0x7f81d40b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:57.928 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.925+0000 7f81d8bac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81d414a580 0x7f81d40b3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:57.928 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.925+0000 7f81d8bac640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f81d414a990 0x7f81d40afbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:57.928 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.925+0000 7f81d8bac640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f81d406e9a0 con 0x7f81d414a580 2024-04-25T19:45:57.928 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.925+0000 7f81d8bac640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f81d406df10 con 0x7f81d40abeb0 2024-04-25T19:45:57.928 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.925+0000 7f81d8bac640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f81d406eb20 con 0x7f81d414a990 2024-04-25T19:45:57.928 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.925+0000 7f81cbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81d414a580 0x7f81d40b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:57.928 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.925+0000 7f81cbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81d414a580 0x7f81d40b3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51102/0 (socket says 172.21.15.29:51102) 2024-04-25T19:45:57.928 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.925+0000 7f81cbfff640 1 -- 172.21.15.29:0/984806072 learned_addr learned my addr 172.21.15.29:0/984806072 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:45:57.929 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.925+0000 7f81d2d9d640 1 --2- 172.21.15.29:0/984806072 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f81d414a990 0x7f81d40afbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:57.929 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.925+0000 7f81d259c640 1 --2- 172.21.15.29:0/984806072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81d40abeb0 0x7f81d40b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:57.929 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.926+0000 7f81d2d9d640 1 -- 172.21.15.29:0/984806072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81d40abeb0 msgr2=0x7f81d40b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:57.929 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.926+0000 7f81d2d9d640 1 --2- 172.21.15.29:0/984806072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81d40abeb0 0x7f81d40b8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:57.929 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.926+0000 7f81d2d9d640 1 -- 172.21.15.29:0/984806072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81d414a580 msgr2=0x7f81d40b3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:57.929 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.926+0000 7f81d2d9d640 1 --2- 172.21.15.29:0/984806072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81d414a580 0x7f81d40b3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:57.929 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.926+0000 7f81d2d9d640 1 -- 172.21.15.29:0/984806072 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d40b2f70 con 0x7f81d414a990 2024-04-25T19:45:57.929 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.926+0000 7f81cbfff640 1 --2- 172.21.15.29:0/984806072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81d414a580 0x7f81d40b3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:45:57.929 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.926+0000 7f81d2d9d640 1 --2- 172.21.15.29:0/984806072 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f81d414a990 0x7f81d40afbf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00c750 tx=0x7f81cc00cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:45:57.929 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.926+0000 7f81caffd640 1 -- 172.21.15.29:0/984806072 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81cc018070 con 0x7f81d414a990 2024-04-25T19:45:57.929 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.926+0000 7f81caffd640 1 -- 172.21.15.29:0/984806072 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81cc004d60 con 0x7f81d414a990 2024-04-25T19:45:57.929 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.926+0000 7f81caffd640 1 -- 172.21.15.29:0/984806072 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81cc00a820 con 0x7f81d414a990 2024-04-25T19:45:57.929 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.926+0000 7f81d8bac640 1 -- 172.21.15.29:0/984806072 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81d40b9820 con 0x7f81d414a990 2024-04-25T19:45:57.930 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.926+0000 7f81d8bac640 1 -- 172.21.15.29:0/984806072 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f81d40b3cb0 con 0x7f81d414a990 2024-04-25T19:45:57.930 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.927+0000 7f81d8bac640 1 -- 172.21.15.29:0/984806072 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81d40692d0 con 0x7f81d414a990 2024-04-25T19:45:57.931 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.929+0000 7f81caffd640 1 -- 172.21.15.29:0/984806072 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 10) ==== 144918+0+0 (secure 0 0 0) 0x7f81cc01b070 con 0x7f81d414a990 2024-04-25T19:45:57.931 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.929+0000 7f81caffd640 20 mgrc handle_mgr_map mgrmap(e 10) 2024-04-25T19:45:57.931 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.929+0000 7f81caffd640 4 mgrc handle_mgr_map Got map version 10 2024-04-25T19:45:57.931 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.929+0000 7f81caffd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] 2024-04-25T19:45:57.931 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.929+0000 7f81caffd640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] 2024-04-25T19:45:57.931 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.929+0000 7f81caffd640 1 --2- 172.21.15.29:0/984806072 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f81b40ac640 0x7f81b40aeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:57.931 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.930+0000 7f81caffd640 1 -- 172.21.15.29:0/984806072 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5137+0+0 (secure 0 0 0) 0x7f81cc0d8d80 con 0x7f81d414a990 2024-04-25T19:45:57.931 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.930+0000 7f81d259c640 1 --2- 172.21.15.29:0/984806072 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f81b40ac640 0x7f81b40aeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:57.932 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.930+0000 7f81d259c640 1 --2- 172.21.15.29:0/984806072 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f81b40ac640 0x7f81b40aeb50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f81c4036000 tx=0x7f81c403c040 comp rx=0 tx=0).ready entity=mgr.4558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:45:57.934 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:57.933+0000 7f81caffd640 1 -- 172.21.15.29:0/984806072 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f81d40ac2c0 con 0x7f81d414a990 2024-04-25T19:45:58.132 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.131+0000 7f81d8bac640 1 -- 172.21.15.29:0/984806072 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd pool ls"} v 0) -- 0x7f81d40ae9b0 con 0x7f81d414a990 2024-04-25T19:45:58.132 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.131+0000 7f81caffd640 1 -- 172.21.15.29:0/984806072 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool ls"}]=0 v26) ==== 59+0+55 (secure 0 0 0) 0x7f81cc0a3080 con 0x7f81d414a990 2024-04-25T19:45:58.243 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.242+0000 7f81d8bac640 1 -- 172.21.15.29:0/984806072 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd pool ls"} v 0) -- 0x7f81d40b11d0 con 0x7f81d414a990 2024-04-25T19:45:58.243 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.242+0000 7f81caffd640 1 -- 172.21.15.29:0/984806072 <== mon.2 v2:172.21.15.29:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool ls"}]=0 v26) ==== 59+0+55 (secure 0 0 0) 0x7f81cc0a7380 con 0x7f81d414a990 2024-04-25T19:45:58.246 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.244+0000 7f81d8bac640 10 mgrc shutdown 2024-04-25T19:45:58.246 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.244+0000 7f81d8bac640 1 -- 172.21.15.29:0/984806072 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f81b40ac640 msgr2=0x7f81b40aeb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:58.246 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.244+0000 7f81d8bac640 1 --2- 172.21.15.29:0/984806072 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f81b40ac640 0x7f81b40aeb50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f81c4036000 tx=0x7f81c403c040 comp rx=0 tx=0).stop 2024-04-25T19:45:58.246 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.244+0000 7f81d8bac640 1 -- 172.21.15.29:0/984806072 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f81d414a990 msgr2=0x7f81d40afbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:58.246 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.244+0000 7f81d8bac640 1 --2- 172.21.15.29:0/984806072 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f81d414a990 0x7f81d40afbf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00c750 tx=0x7f81cc00cc10 comp rx=0 tx=0).stop 2024-04-25T19:45:58.246 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.244+0000 7f81d8bac640 1 -- 172.21.15.29:0/984806072 shutdown_connections 2024-04-25T19:45:58.246 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.244+0000 7f81d8bac640 1 --2- 172.21.15.29:0/984806072 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f81b40ac640 0x7f81b40aeb50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:58.246 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.244+0000 7f81d8bac640 1 --2- 172.21.15.29:0/984806072 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f81d414a990 0x7f81d40afbf0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:58.246 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.244+0000 7f81d8bac640 1 --2- 172.21.15.29:0/984806072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81d414a580 0x7f81d40b3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:58.246 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.244+0000 7f81d8bac640 1 --2- 172.21.15.29:0/984806072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81d40abeb0 0x7f81d40b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:58.247 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.244+0000 7f81d8bac640 1 -- 172.21.15.29:0/984806072 >> 172.21.15.29:0/984806072 conn(0x7f81d40bc870 msgr2=0x7f81d40aade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:45:58.247 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.244+0000 7f81d8bac640 1 -- 172.21.15.29:0/984806072 shutdown_connections 2024-04-25T19:45:58.247 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.244+0000 7f81d8bac640 1 -- 172.21.15.29:0/984806072 wait complete. 2024-04-25T19:45:58.254 INFO:tasks.workunit.client.0.smithi029.stdout:by-host-smithi029 2024-04-25T19:45:58.255 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph mgr module disable localpool 2024-04-25T19:45:58.347 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.346+0000 7f1e9bd4c640 1 Processor -- start 2024-04-25T19:45:58.347 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.346+0000 7f1e9bd4c640 1 -- start start 2024-04-25T19:45:58.349 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.347+0000 7f1e9bd4c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1e940ae940 0x7f1e9406e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:58.349 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.347+0000 7f1e9bd4c640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1e940abe00 0x7f1e94151620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:58.349 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.347+0000 7f1e9bd4c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e940aed50 con 0x7f1e940ab9f0 2024-04-25T19:45:58.349 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.347+0000 7f1e9bd4c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f1e9406e2a0 con 0x7f1e940ae940 2024-04-25T19:45:58.349 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.347+0000 7f1e9bd4c640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f1e9406d4f0 con 0x7f1e940abe00 2024-04-25T19:45:58.349 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.347+0000 7f1e99ae8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1e940ae940 0x7f1e9406e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:58.349 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.347+0000 7f1e992e7640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1e940ab9f0 0x7f1e94146500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43216/0 (socket says 172.21.15.29:43216) 2024-04-25T19:45:58.349 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.347+0000 7f1e992e7640 1 -- 172.21.15.29:0/3788625298 learned_addr learned my addr 172.21.15.29:0/3788625298 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:45:58.349 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.348+0000 7f1e9a2e9640 1 --2- 172.21.15.29:0/3788625298 >> v2:172.21.15.102:3300/0 conn(0x7f1e940abe00 0x7f1e94151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:58.349 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.348+0000 7f1e99ae8640 1 -- 172.21.15.29:0/3788625298 >> v2:172.21.15.102:3300/0 conn(0x7f1e940abe00 msgr2=0x7f1e94151620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:45:58.350 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.348+0000 7f1e99ae8640 1 --2- 172.21.15.29:0/3788625298 >> v2:172.21.15.102:3300/0 conn(0x7f1e940abe00 0x7f1e94151620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:58.350 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.348+0000 7f1e99ae8640 1 -- 172.21.15.29:0/3788625298 >> v1:172.21.15.29:6789/0 conn(0x7f1e940ab9f0 legacy=0x7f1e94146500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:58.350 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.348+0000 7f1e98ae6640 1 -- 172.21.15.29:0/3788625298 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2565879776 0 0) 0x7f1e88001110 con 0x7f1e940ab9f0 2024-04-25T19:45:58.350 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.348+0000 7f1e99ae8640 1 -- 172.21.15.29:0/3788625298 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e94146c10 con 0x7f1e940ae940 2024-04-25T19:45:58.351 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.348+0000 7f1e99ae8640 1 --2- 172.21.15.29:0/3788625298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e940ae940 0x7f1e9406e710 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1e9406dcb0 tx=0x7f1e8c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db75ce9738c8afe5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:45:58.351 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.349+0000 7f1e98ae6640 1 -- 172.21.15.29:0/3788625298 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c035020 con 0x7f1e940ae940 2024-04-25T19:45:58.351 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.349+0000 7f1e98ae6640 1 -- 172.21.15.29:0/3788625298 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e8c031830 con 0x7f1e940ae940 2024-04-25T19:45:58.352 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.349+0000 7f1e98ae6640 1 -- 172.21.15.29:0/3788625298 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c031c00 con 0x7f1e940ae940 2024-04-25T19:45:58.352 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.349+0000 7f1e9bd4c640 1 -- 172.21.15.29:0/3788625298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e940ae940 msgr2=0x7f1e9406e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:58.352 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.349+0000 7f1e9bd4c640 1 --2- 172.21.15.29:0/3788625298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e940ae940 0x7f1e9406e710 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1e9406dcb0 tx=0x7f1e8c02b8d0 comp rx=0 tx=0).stop 2024-04-25T19:45:58.352 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.349+0000 7f1e9bd4c640 1 -- 172.21.15.29:0/3788625298 shutdown_connections 2024-04-25T19:45:58.352 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.349+0000 7f1e9bd4c640 1 --2- 172.21.15.29:0/3788625298 >> v2:172.21.15.102:3300/0 conn(0x7f1e940abe00 0x7f1e94151620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:58.352 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.349+0000 7f1e9bd4c640 1 --2- 172.21.15.29:0/3788625298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e940ae940 0x7f1e9406e710 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:58.352 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.349+0000 7f1e9bd4c640 1 -- 172.21.15.29:0/3788625298 >> 172.21.15.29:0/3788625298 conn(0x7f1e940bc5c0 msgr2=0x7f1e940bc9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:45:58.352 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.349+0000 7f1e9bd4c640 1 -- 172.21.15.29:0/3788625298 shutdown_connections 2024-04-25T19:45:58.352 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.349+0000 7f1e9bd4c640 1 -- 172.21.15.29:0/3788625298 wait complete. 2024-04-25T19:45:58.354 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.351+0000 7f1e9bd4c640 1 Processor -- start 2024-04-25T19:45:58.354 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.351+0000 7f1e9bd4c640 1 -- start start 2024-04-25T19:45:58.355 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.352+0000 7f1e9bd4c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1e940ab9f0 0x7f1e9413db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:58.355 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.352+0000 7f1e9bd4c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e940abe00 0x7f1e941e11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:58.355 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.352+0000 7f1e9bd4c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e940ae940 0x7f1e9413ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:58.355 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.352+0000 7f1e9bd4c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9406da00 con 0x7f1e940ae940 2024-04-25T19:45:58.355 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.352+0000 7f1e9bd4c640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9406d740 con 0x7f1e940abe00 2024-04-25T19:45:58.355 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.352+0000 7f1e9bd4c640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f1e94151b60 con 0x7f1e940ab9f0 2024-04-25T19:45:58.355 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.352+0000 7f1e99ae8640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1e940ab9f0 0x7f1e9413db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:58.355 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.352+0000 7f1e99ae8640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1e940ab9f0 0x7f1e9413db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:49834/0 (socket says 172.21.15.29:49834) 2024-04-25T19:45:58.355 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.352+0000 7f1e99ae8640 1 -- 172.21.15.29:0/4185708670 learned_addr learned my addr 172.21.15.29:0/4185708670 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:45:58.355 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.352+0000 7f1e9a2e9640 1 --2- 172.21.15.29:0/4185708670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e940ae940 0x7f1e9413ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:58.355 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.352+0000 7f1e992e7640 1 --2- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e940abe00 0x7f1e941e11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:58.355 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.352+0000 7f1e99ae8640 1 -- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e940abe00 msgr2=0x7f1e941e11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:58.355 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.352+0000 7f1e99ae8640 1 --2- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e940abe00 0x7f1e941e11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:58.355 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.352+0000 7f1e99ae8640 1 -- 172.21.15.29:0/4185708670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e940ae940 msgr2=0x7f1e9413ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:45:58.356 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.352+0000 7f1e99ae8640 1 --2- 172.21.15.29:0/4185708670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e940ae940 0x7f1e9413ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:58.356 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.352+0000 7f1e99ae8640 1 -- 172.21.15.29:0/4185708670 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e9413e4e0 con 0x7f1e940ab9f0 2024-04-25T19:45:58.356 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.352+0000 7f1e99ae8640 1 --2- 172.21.15.29:0/4185708670 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1e940ab9f0 0x7f1e9413db60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c034040 tx=0x7f1e8c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:45:58.356 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.353+0000 7f1e927fc640 1 -- 172.21.15.29:0/4185708670 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c035040 con 0x7f1e940ab9f0 2024-04-25T19:45:58.356 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.353+0000 7f1e927fc640 1 -- 172.21.15.29:0/4185708670 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e8c004230 con 0x7f1e940ab9f0 2024-04-25T19:45:58.356 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.353+0000 7f1e9bd4c640 1 -- 172.21.15.29:0/4185708670 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e94073a90 con 0x7f1e940ab9f0 2024-04-25T19:45:58.356 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.353+0000 7f1e927fc640 1 -- 172.21.15.29:0/4185708670 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c03a700 con 0x7f1e940ab9f0 2024-04-25T19:45:58.356 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.353+0000 7f1e9bd4c640 1 -- 172.21.15.29:0/4185708670 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e9413e0a0 con 0x7f1e940ab9f0 2024-04-25T19:45:58.356 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.354+0000 7f1e9bd4c640 1 -- 172.21.15.29:0/4185708670 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e940692d0 con 0x7f1e940ab9f0 2024-04-25T19:45:58.357 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.356+0000 7f1e927fc640 1 -- 172.21.15.29:0/4185708670 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 10) ==== 144918+0+0 (secure 0 0 0) 0x7f1e8c038020 con 0x7f1e940ab9f0 2024-04-25T19:45:58.357 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.356+0000 7f1e927fc640 20 mgrc handle_mgr_map mgrmap(e 10) 2024-04-25T19:45:58.357 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.356+0000 7f1e927fc640 4 mgrc handle_mgr_map Got map version 10 2024-04-25T19:45:58.357 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.356+0000 7f1e927fc640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] 2024-04-25T19:45:58.358 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.356+0000 7f1e927fc640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] 2024-04-25T19:45:58.358 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.356+0000 7f1e927fc640 1 --2- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f1e800ac610 0x7f1e800aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:58.358 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.357+0000 7f1e927fc640 1 -- 172.21.15.29:0/4185708670 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 5137+0+0 (secure 0 0 0) 0x7f1e8c0fb950 con 0x7f1e940ab9f0 2024-04-25T19:45:58.358 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.357+0000 7f1e992e7640 1 --2- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f1e800ac610 0x7f1e800aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:45:58.358 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.357+0000 7f1e992e7640 1 --2- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f1e800ac610 0x7f1e800aeb20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1e941e1100 tx=0x7f1e88009040 comp rx=0 tx=0).ready entity=mgr.4558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:45:58.361 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.360+0000 7f1e927fc640 1 -- 172.21.15.29:0/4185708670 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1e8c053080 con 0x7f1e940ab9f0 2024-04-25T19:45:58.536 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:45:58.539 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:45:58.596 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:58.595+0000 7f1e9bd4c640 1 -- 172.21.15.29:0/4185708670 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "mgr module disable", "module": "localpool"} v 0) -- 0x7f1e940b3160 con 0x7f1e940ab9f0 2024-04-25T19:45:59.085 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.084+0000 7f1e927fc640 1 -- 172.21.15.29:0/4185708670 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mgrmap(e 11) ==== 144961+0+0 (secure 0 0 0) 0x7f1e8c0c1da0 con 0x7f1e940ab9f0 2024-04-25T19:45:59.086 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.084+0000 7f1e927fc640 20 mgrc handle_mgr_map mgrmap(e 11) 2024-04-25T19:45:59.086 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.084+0000 7f1e927fc640 4 mgrc handle_mgr_map Got map version 11 2024-04-25T19:45:59.086 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.084+0000 7f1e927fc640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] 2024-04-25T19:45:59.086 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.085+0000 7f1e927fc640 1 -- 172.21.15.29:0/4185708670 <== mon.2 v2:172.21.15.29:3301/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "localpool"}]=0 v11) ==== 89+0+0 (secure 0 0 0) 0x7f1e940b3160 con 0x7f1e940ab9f0 2024-04-25T19:45:59.114 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.112+0000 7f1e992e7640 1 -- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f1e800ac610 msgr2=0x7f1e800aeb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-25T19:45:59.114 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.112+0000 7f1e992e7640 1 -- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f1e800ac610 msgr2=0x7f1e800aeb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T19:45:59.114 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.112+0000 7f1e992e7640 1 --2- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f1e800ac610 0x7f1e800aeb20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1e941e1100 tx=0x7f1e88009040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T19:45:59.115 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.112+0000 7f1e992e7640 1 --2- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f1e800ac610 0x7f1e800aeb20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1e941e1100 tx=0x7f1e88009040 comp rx=0 tx=0).stop 2024-04-25T19:45:59.115 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.112+0000 7f1e927fc640 4 mgrc ms_handle_reset ms_handle_reset con 0x7f1e800ac610 2024-04-25T19:45:59.115 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.112+0000 7f1e927fc640 4 mgrc reconnect Terminating session with v2:172.21.15.102:6824/1500823528 2024-04-25T19:45:59.115 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.112+0000 7f1e927fc640 1 -- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f1e800ac610 msgr2=0x7f1e800aeb20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T19:45:59.115 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.112+0000 7f1e927fc640 1 --2- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f1e800ac610 0x7f1e800aeb20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:45:59.115 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.112+0000 7f1e927fc640 4 mgrc reconnect waiting to retry connect until 594.457323s 2024-04-25T19:45:59.225 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:59.224+0000 7f33fe212040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T19:45:59.233 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:59.232+0000 7f4e1e83c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T19:45:59.234 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:59.232+0000 7f846c637040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T19:45:59.248 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.247+0000 7f1e9bd4c640 1 -- 172.21.15.29:0/4185708670 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "mgr module disable", "module": "localpool"} v 0) -- 0x7f1e780050a0 con 0x7f1e940ab9f0 2024-04-25T19:45:59.343 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:59.342+0000 7f33fe212040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:45:59.347 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:59.346+0000 7f846c637040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:45:59.348 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:59.346+0000 7f4e1e83c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:45:59.358 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.356+0000 7f1e90ff9640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] 2024-04-25T19:45:59.358 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.356+0000 7f1e90ff9640 1 --2- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f1e800ac610 0x7f1e941e1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:45:59.358 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.357+0000 7f1e992e7640 1 -- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f1e800ac610 msgr2=0x7f1e941e1700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6824/1500823528 2024-04-25T19:45:59.358 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.357+0000 7f1e992e7640 1 --2- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f1e800ac610 0x7f1e941e1700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T19:45:59.393 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:59.391+0000 7f33fe212040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T19:45:59.397 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:59.395+0000 7f846c637040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T19:45:59.397 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:59.395+0000 7f4e1e83c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T19:45:59.559 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.558+0000 7f1e992e7640 1 -- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f1e800ac610 msgr2=0x7f1e941e1700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6824/1500823528 2024-04-25T19:45:59.559 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.558+0000 7f1e992e7640 1 --2- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f1e800ac610 0x7f1e941e1700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T19:45:59.603 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:59.602+0000 7f4e1e83c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T19:45:59.603 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:59.602+0000 7f846c637040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T19:45:59.605 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:59.605+0000 7f33fe212040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T19:45:59.647 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:45:59.646+0000 7f4e1e83c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T19:45:59.647 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:45:59.646+0000 7f846c637040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T19:45:59.650 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:45:59.649+0000 7f33fe212040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T19:45:59.960 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.959+0000 7f1e992e7640 1 -- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f1e800ac610 msgr2=0x7f1e941e1700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6824/1500823528 2024-04-25T19:45:59.960 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:45:59.959+0000 7f1e992e7640 1 --2- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f1e800ac610 0x7f1e941e1700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T19:46:00.085 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.084+0000 7f1e927fc640 1 -- 172.21.15.29:0/4185708670 <== mon.2 v2:172.21.15.29:3301/0 9 ==== mgrmap(e 12) ==== 144961+0+0 (secure 0 0 0) 0x7f1e8c0fddb0 con 0x7f1e940ab9f0 2024-04-25T19:46:00.086 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.084+0000 7f1e927fc640 20 mgrc handle_mgr_map mgrmap(e 12) 2024-04-25T19:46:00.086 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.085+0000 7f1e927fc640 4 mgrc handle_mgr_map Got map version 12 2024-04-25T19:46:00.086 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.085+0000 7f1e927fc640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] 2024-04-25T19:46:00.086 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.086+0000 7f1e927fc640 1 -- 172.21.15.29:0/4185708670 <== mon.2 v2:172.21.15.29:3301/0 10 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "localpool"}]=0 module 'localpool' is already disabled v12) ==== 127+0+0 (secure 0 0 0) 0x7f1e94146c10 con 0x7f1e940ab9f0 2024-04-25T19:46:00.087 INFO:tasks.workunit.client.0.smithi029.stderr:module 'localpool' is already disabled 2024-04-25T19:46:00.090 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.089+0000 7f1e9bd4c640 10 mgrc shutdown 2024-04-25T19:46:00.090 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.089+0000 7f1e9bd4c640 1 -- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f1e800ac610 msgr2=0x7f1e941e1700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T19:46:00.090 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.089+0000 7f1e9bd4c640 1 --2- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f1e800ac610 0x7f1e941e1700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:00.090 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.089+0000 7f1e9bd4c640 1 -- 172.21.15.29:0/4185708670 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1e940ab9f0 msgr2=0x7f1e9413db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:00.090 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.089+0000 7f1e9bd4c640 1 --2- 172.21.15.29:0/4185708670 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1e940ab9f0 0x7f1e9413db60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c034040 tx=0x7f1e8c0046c0 comp rx=0 tx=0).stop 2024-04-25T19:46:00.091 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.089+0000 7f1e9bd4c640 1 -- 172.21.15.29:0/4185708670 shutdown_connections 2024-04-25T19:46:00.091 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.089+0000 7f1e9bd4c640 1 --2- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f1e800ac610 0x7f1e941e1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:00.091 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.089+0000 7f1e9bd4c640 1 --2- 172.21.15.29:0/4185708670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e940ae940 0x7f1e9413ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:00.091 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.089+0000 7f1e9bd4c640 1 --2- 172.21.15.29:0/4185708670 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e940abe00 0x7f1e941e11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:00.091 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.089+0000 7f1e9bd4c640 1 --2- 172.21.15.29:0/4185708670 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1e940ab9f0 0x7f1e9413db60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:00.091 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.089+0000 7f1e9bd4c640 1 -- 172.21.15.29:0/4185708670 >> 172.21.15.29:0/4185708670 conn(0x7f1e940bc5c0 msgr2=0x7f1e94146df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:00.091 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.089+0000 7f1e9bd4c640 1 -- 172.21.15.29:0/4185708670 shutdown_connections 2024-04-25T19:46:00.091 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.089+0000 7f1e9bd4c640 1 -- 172.21.15.29:0/4185708670 wait complete. 2024-04-25T19:46:00.100 INFO:tasks.workunit.client.0.smithi029.stderr:++ ceph osd pool ls 2024-04-25T19:46:00.100 INFO:tasks.workunit.client.0.smithi029.stderr:++ grep '^by-host-' 2024-04-25T19:46:00.174 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:46:00.173+0000 7f33fe212040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T19:46:00.191 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.190+0000 7ff2ac329640 1 Processor -- start 2024-04-25T19:46:00.192 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.191+0000 7ff2ac329640 1 -- start start 2024-04-25T19:46:00.193 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.192+0000 7ff2ac329640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff2a40a5d70 0x7ff2a40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:00.194 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.192+0000 7ff2ac329640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2a40b4a90 con 0x7ff2a40a66c0 2024-04-25T19:46:00.194 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.192+0000 7ff2ac329640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2a40b4d30 con 0x7ff2a406d0f0 2024-04-25T19:46:00.194 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.192+0000 7ff2ac329640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7ff2a40b4180 con 0x7ff2a40a5d70 2024-04-25T19:46:00.194 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.192+0000 7ff2a98c4640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff2a40a5d70 0x7ff2a40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:00.194 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.192+0000 7ff2aa8c6640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff2a40a66c0 0x7ff2a4151fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43248/0 (socket says 172.21.15.29:43248) 2024-04-25T19:46:00.194 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.192+0000 7ff2aa8c6640 1 -- 172.21.15.29:0/2274089651 learned_addr learned my addr 172.21.15.29:0/2274089651 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:46:00.194 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.192+0000 7ff2a90c3640 1 -- 172.21.15.29:0/2274089651 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1013243702 0 0) 0x7ff2a40b4a90 con 0x7ff2a40a66c0 2024-04-25T19:46:00.194 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.192+0000 7ff2a98c4640 1 -- 172.21.15.29:0/2274089651 >> v1:172.21.15.102:6789/0 conn(0x7ff2a406d0f0 legacy=0x7ff2a40b5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:00.194 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.192+0000 7ff2a98c4640 1 -- 172.21.15.29:0/2274089651 >> v1:172.21.15.29:6789/0 conn(0x7ff2a40a66c0 legacy=0x7ff2a4151fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:00.194 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.192+0000 7ff2a98c4640 1 -- 172.21.15.29:0/2274089651 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2a40b4300 con 0x7ff2a40a5d70 2024-04-25T19:46:00.195 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.193+0000 7ff2a98c4640 1 --2- 172.21.15.29:0/2274089651 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff2a40a5d70 0x7ff2a40a6180 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff2980097d0 tx=0x7ff29802b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ce95a85c4c822a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:00.195 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.193+0000 7ff2a90c3640 1 -- 172.21.15.29:0/2274089651 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff298031770 con 0x7ff2a40a5d70 2024-04-25T19:46:00.196 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.193+0000 7ff2a90c3640 1 -- 172.21.15.29:0/2274089651 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff298037070 con 0x7ff2a40a5d70 2024-04-25T19:46:00.196 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.193+0000 7ff2a90c3640 1 -- 172.21.15.29:0/2274089651 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff298031b30 con 0x7ff2a40a5d70 2024-04-25T19:46:00.196 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.194+0000 7ff2ac329640 1 -- 172.21.15.29:0/2274089651 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff2a40a5d70 msgr2=0x7ff2a40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:00.196 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.194+0000 7ff2ac329640 1 --2- 172.21.15.29:0/2274089651 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff2a40a5d70 0x7ff2a40a6180 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff2980097d0 tx=0x7ff29802b810 comp rx=0 tx=0).stop 2024-04-25T19:46:00.196 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.194+0000 7ff2ac329640 1 -- 172.21.15.29:0/2274089651 shutdown_connections 2024-04-25T19:46:00.196 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.194+0000 7ff2ac329640 1 --2- 172.21.15.29:0/2274089651 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff2a40a5d70 0x7ff2a40a6180 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:00.196 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.194+0000 7ff2ac329640 1 -- 172.21.15.29:0/2274089651 >> 172.21.15.29:0/2274089651 conn(0x7ff2a406df60 msgr2=0x7ff2a406e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:00.196 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.194+0000 7ff2ac329640 1 -- 172.21.15.29:0/2274089651 shutdown_connections 2024-04-25T19:46:00.197 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.194+0000 7ff2ac329640 1 -- 172.21.15.29:0/2274089651 wait complete. 2024-04-25T19:46:00.200 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.196+0000 7ff2ac329640 1 Processor -- start 2024-04-25T19:46:00.200 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.196+0000 7ff2ac329640 1 -- start start 2024-04-25T19:46:00.200 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.197+0000 7ff2ac329640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff2a406d0f0 0x7ff2a41d1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:00.200 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.197+0000 7ff2ac329640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a40a5d70 0x7ff2a40b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:00.200 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.197+0000 7ff2ac329640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2a40a66c0 0x7ff2a41dd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:00.200 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.197+0000 7ff2ac329640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a40a6c50 con 0x7ff2a40a5d70 2024-04-25T19:46:00.200 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.197+0000 7ff2ac329640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a40a6ad0 con 0x7ff2a40a66c0 2024-04-25T19:46:00.200 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.197+0000 7ff2ac329640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7ff2a40b2cd0 con 0x7ff2a406d0f0 2024-04-25T19:46:00.201 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.197+0000 7ff2a98c4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a40a5d70 0x7ff2a40b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:00.201 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.197+0000 7ff2aa0c5640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff2a406d0f0 0x7ff2a41d1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:00.201 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.197+0000 7ff2aa0c5640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff2a406d0f0 0x7ff2a41d1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:49882/0 (socket says 172.21.15.29:49882) 2024-04-25T19:46:00.201 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.197+0000 7ff2a98c4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a40a5d70 0x7ff2a40b17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51142/0 (socket says 172.21.15.29:51142) 2024-04-25T19:46:00.201 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.197+0000 7ff2aa0c5640 1 -- 172.21.15.29:0/4006713059 learned_addr learned my addr 172.21.15.29:0/4006713059 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:46:00.201 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.197+0000 7ff2aa8c6640 1 --2- 172.21.15.29:0/4006713059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2a40a66c0 0x7ff2a41dd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:00.201 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.197+0000 7ff2aa0c5640 1 -- 172.21.15.29:0/4006713059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2a40a66c0 msgr2=0x7ff2a41dd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:00.201 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.197+0000 7ff2aa0c5640 1 --2- 172.21.15.29:0/4006713059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2a40a66c0 0x7ff2a41dd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:00.201 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.197+0000 7ff2aa0c5640 1 -- 172.21.15.29:0/4006713059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a40a5d70 msgr2=0x7ff2a40b17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:00.201 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.197+0000 7ff2aa0c5640 1 --2- 172.21.15.29:0/4006713059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a40a5d70 0x7ff2a40b17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:00.201 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.197+0000 7ff2aa0c5640 1 -- 172.21.15.29:0/4006713059 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2a40b1cf0 con 0x7ff2a406d0f0 2024-04-25T19:46:00.201 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.197+0000 7ff2a98c4640 1 --2- 172.21.15.29:0/4006713059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a40a5d70 0x7ff2a40b17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:46:00.201 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.197+0000 7ff2aa0c5640 1 --2- 172.21.15.29:0/4006713059 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff2a406d0f0 0x7ff2a41d1960 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff2a000c430 tx=0x7ff2a000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:00.201 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.197+0000 7ff296ffd640 1 -- 172.21.15.29:0/4006713059 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a0017070 con 0x7ff2a406d0f0 2024-04-25T19:46:00.202 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.198+0000 7ff296ffd640 1 -- 172.21.15.29:0/4006713059 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2a00052d0 con 0x7ff2a406d0f0 2024-04-25T19:46:00.202 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.198+0000 7ff2ac329640 1 -- 172.21.15.29:0/4006713059 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2a409aa50 con 0x7ff2a406d0f0 2024-04-25T19:46:00.202 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.198+0000 7ff296ffd640 1 -- 172.21.15.29:0/4006713059 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a0003830 con 0x7ff2a406d0f0 2024-04-25T19:46:00.202 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.198+0000 7ff2ac329640 1 -- 172.21.15.29:0/4006713059 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2a4147810 con 0x7ff2a406d0f0 2024-04-25T19:46:00.202 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.199+0000 7ff2ac329640 1 -- 172.21.15.29:0/4006713059 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2a40692d0 con 0x7ff2a406d0f0 2024-04-25T19:46:00.202 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:00.198+0000 7f4e1e83c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T19:46:00.203 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:46:00.198+0000 7f846c637040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T19:46:00.203 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.200+0000 7ff296ffd640 1 -- 172.21.15.29:0/4006713059 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 12) ==== 144961+0+0 (secure 0 0 0) 0x7ff2a001a070 con 0x7ff2a406d0f0 2024-04-25T19:46:00.204 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.200+0000 7ff296ffd640 20 mgrc handle_mgr_map mgrmap(e 12) 2024-04-25T19:46:00.204 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.201+0000 7ff296ffd640 4 mgrc handle_mgr_map Got map version 12 2024-04-25T19:46:00.204 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.201+0000 7ff296ffd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] 2024-04-25T19:46:00.204 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.201+0000 7ff296ffd640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] 2024-04-25T19:46:00.204 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.201+0000 7ff296ffd640 1 --2- 172.21.15.29:0/4006713059 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7ff2880ac650 0x7ff2880aeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:00.204 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.201+0000 7ff2a98c4640 1 -- 172.21.15.29:0/4006713059 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7ff2880ac650 msgr2=0x7ff2880aeb60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6824/1500823528 2024-04-25T19:46:00.204 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.201+0000 7ff296ffd640 1 -- 172.21.15.29:0/4006713059 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 5566+0+0 (secure 0 0 0) 0x7ff2a00d9010 con 0x7ff2a406d0f0 2024-04-25T19:46:00.204 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.201+0000 7ff2a98c4640 1 --2- 172.21.15.29:0/4006713059 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7ff2880ac650 0x7ff2880aeb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T19:46:00.205 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.204+0000 7ff296ffd640 1 -- 172.21.15.29:0/4006713059 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff2a0021020 con 0x7ff2a406d0f0 2024-04-25T19:46:00.265 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:46:00.264+0000 7f33fe212040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T19:46:00.286 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:00.286+0000 7f4e1e83c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T19:46:00.287 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:46:00.286+0000 7f846c637040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T19:46:00.309 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:46:00.308+0000 7f33fe212040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T19:46:00.330 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:00.329+0000 7f4e1e83c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T19:46:00.330 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:46:00.329+0000 7f846c637040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T19:46:00.391 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.390+0000 7ff2ac329640 1 -- 172.21.15.29:0/4006713059 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd pool ls"} v 0) -- 0x7ff2a406dd40 con 0x7ff2a406d0f0 2024-04-25T19:46:00.392 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.391+0000 7ff296ffd640 1 -- 172.21.15.29:0/4006713059 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool ls"}]=0 v28) ==== 59+0+73 (secure 0 0 0) 0x7ff2a00a3050 con 0x7ff2a406d0f0 2024-04-25T19:46:00.403 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.402+0000 7ff2a98c4640 1 -- 172.21.15.29:0/4006713059 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7ff2880ac650 msgr2=0x7ff2880aeb60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6824/1500823528 2024-04-25T19:46:00.403 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.402+0000 7ff2a98c4640 1 --2- 172.21.15.29:0/4006713059 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7ff2880ac650 0x7ff2880aeb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T19:46:00.481 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:46:00.480+0000 7f33fe212040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T19:46:00.499 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:00.499+0000 7f4e1e83c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T19:46:00.500 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:46:00.499+0000 7f846c637040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T19:46:00.501 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.500+0000 7ff2ac329640 1 -- 172.21.15.29:0/4006713059 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd pool ls"} v 0) -- 0x7ff2a40b6470 con 0x7ff2a406d0f0 2024-04-25T19:46:00.502 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.501+0000 7ff296ffd640 1 -- 172.21.15.29:0/4006713059 <== mon.2 v2:172.21.15.29:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool ls"}]=0 v28) ==== 59+0+73 (secure 0 0 0) 0x7ff2a00a7350 con 0x7ff2a406d0f0 2024-04-25T19:46:00.505 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.503+0000 7ff2ac329640 10 mgrc shutdown 2024-04-25T19:46:00.505 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.503+0000 7ff2ac329640 1 -- 172.21.15.29:0/4006713059 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7ff2880ac650 msgr2=0x7ff2880aeb60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T19:46:00.505 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.503+0000 7ff2ac329640 1 --2- 172.21.15.29:0/4006713059 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7ff2880ac650 0x7ff2880aeb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:00.505 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.503+0000 7ff2ac329640 1 -- 172.21.15.29:0/4006713059 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff2a406d0f0 msgr2=0x7ff2a41d1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:00.505 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.503+0000 7ff2ac329640 1 --2- 172.21.15.29:0/4006713059 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff2a406d0f0 0x7ff2a41d1960 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff2a000c430 tx=0x7ff2a000c8f0 comp rx=0 tx=0).stop 2024-04-25T19:46:00.505 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.503+0000 7ff2ac329640 1 -- 172.21.15.29:0/4006713059 shutdown_connections 2024-04-25T19:46:00.506 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.503+0000 7ff2ac329640 1 --2- 172.21.15.29:0/4006713059 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7ff2880ac650 0x7ff2880aeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:00.506 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.503+0000 7ff2ac329640 1 --2- 172.21.15.29:0/4006713059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2a40a66c0 0x7ff2a41dd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:00.506 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.503+0000 7ff2ac329640 1 --2- 172.21.15.29:0/4006713059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a40a5d70 0x7ff2a40b17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:00.506 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.503+0000 7ff2ac329640 1 --2- 172.21.15.29:0/4006713059 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff2a406d0f0 0x7ff2a41d1960 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:00.506 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.503+0000 7ff2ac329640 1 -- 172.21.15.29:0/4006713059 >> 172.21.15.29:0/4006713059 conn(0x7ff2a406df60 msgr2=0x7ff2a40b9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:00.506 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.503+0000 7ff2ac329640 1 -- 172.21.15.29:0/4006713059 shutdown_connections 2024-04-25T19:46:00.506 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.503+0000 7ff2ac329640 1 -- 172.21.15.29:0/4006713059 wait complete. 2024-04-25T19:46:00.514 INFO:tasks.workunit.client.0.smithi029.stderr:+ for p in `ceph osd pool ls | grep '^by-host-'` 2024-04-25T19:46:00.514 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph osd pool rm by-host-smithi029 by-host-smithi029 --yes-i-really-really-mean-it 2024-04-25T19:46:00.529 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:46:00.528+0000 7f33fe212040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T19:46:00.546 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:00.545+0000 7f4e1e83c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T19:46:00.546 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:46:00.546+0000 7f846c637040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T19:46:00.583 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:46:00.582+0000 7f33fe212040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T19:46:00.595 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:00.595+0000 7f4e1e83c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T19:46:00.596 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:46:00.595+0000 7f846c637040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T19:46:00.609 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.608+0000 7f5b272f9640 1 Processor -- start 2024-04-25T19:46:00.609 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.608+0000 7f5b272f9640 1 -- start start 2024-04-25T19:46:00.611 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.609+0000 7f5b272f9640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b200ab430 0x7f5b200b4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:00.611 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.609+0000 7f5b272f9640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5b2014d490 0x7f5b2014f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:00.611 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.609+0000 7f5b272f9640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b200b4ce0 con 0x7f5b200b4640 2024-04-25T19:46:00.611 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.609+0000 7f5b272f9640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f5b2006cdd0 con 0x7f5b2014d490 2024-04-25T19:46:00.611 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.609+0000 7f5b272f9640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f5b20148100 con 0x7f5b200ab430 2024-04-25T19:46:00.611 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.609+0000 7f5b25095640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b200ab430 0x7f5b200b4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:00.611 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.609+0000 7f5b25095640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b200ab430 0x7f5b200b4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:49888/0 (socket says 172.21.15.29:49888) 2024-04-25T19:46:00.611 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.609+0000 7f5b25095640 1 -- 172.21.15.29:0/2954659524 learned_addr learned my addr 172.21.15.29:0/2954659524 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:46:00.612 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.609+0000 7f5b25896640 1 --2- 172.21.15.29:0/2954659524 >> v2:172.21.15.102:3300/0 conn(0x7f5b2014d490 0x7f5b2014f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:00.612 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.610+0000 7f5b25095640 1 -- 172.21.15.29:0/2954659524 >> v2:172.21.15.102:3300/0 conn(0x7f5b2014d490 msgr2=0x7f5b2014f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:46:00.612 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.610+0000 7f5b25095640 1 --2- 172.21.15.29:0/2954659524 >> v2:172.21.15.102:3300/0 conn(0x7f5b2014d490 0x7f5b2014f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:00.612 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.610+0000 7f5b25095640 1 -- 172.21.15.29:0/2954659524 >> v1:172.21.15.102:6789/0 conn(0x7f5b200b4640 legacy=0x7f5b20148730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:00.612 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.610+0000 7f5b25095640 1 -- 172.21.15.29:0/2954659524 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b200b4fa0 con 0x7f5b200ab430 2024-04-25T19:46:00.613 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.610+0000 7f5b25095640 1 --2- 172.21.15.29:0/2954659524 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b200ab430 0x7f5b200b4100 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009870 tx=0x7f5b1002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efe121910b44af1b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:00.613 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.610+0000 7f5b17fff640 1 -- 172.21.15.29:0/2954659524 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b10034070 con 0x7f5b200ab430 2024-04-25T19:46:00.613 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.611+0000 7f5b17fff640 1 -- 172.21.15.29:0/2954659524 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b10030840 con 0x7f5b200ab430 2024-04-25T19:46:00.613 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.611+0000 7f5b17fff640 1 -- 172.21.15.29:0/2954659524 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b10030c10 con 0x7f5b200ab430 2024-04-25T19:46:00.613 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.611+0000 7f5b272f9640 1 -- 172.21.15.29:0/2954659524 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b200ab430 msgr2=0x7f5b200b4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:00.613 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.611+0000 7f5b272f9640 1 --2- 172.21.15.29:0/2954659524 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b200ab430 0x7f5b200b4100 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009870 tx=0x7f5b1002b980 comp rx=0 tx=0).stop 2024-04-25T19:46:00.614 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.611+0000 7f5b272f9640 1 -- 172.21.15.29:0/2954659524 shutdown_connections 2024-04-25T19:46:00.614 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.611+0000 7f5b272f9640 1 --2- 172.21.15.29:0/2954659524 >> v2:172.21.15.102:3300/0 conn(0x7f5b2014d490 0x7f5b2014f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:00.614 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.611+0000 7f5b272f9640 1 --2- 172.21.15.29:0/2954659524 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b200ab430 0x7f5b200b4100 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:00.614 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.611+0000 7f5b272f9640 1 -- 172.21.15.29:0/2954659524 >> 172.21.15.29:0/2954659524 conn(0x7f5b200b6c60 msgr2=0x7f5b200b7070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:00.614 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.611+0000 7f5b272f9640 1 -- 172.21.15.29:0/2954659524 shutdown_connections 2024-04-25T19:46:00.614 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.612+0000 7f5b272f9640 1 -- 172.21.15.29:0/2954659524 wait complete. 2024-04-25T19:46:00.616 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.613+0000 7f5b272f9640 1 Processor -- start 2024-04-25T19:46:00.616 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.613+0000 7f5b272f9640 1 -- start start 2024-04-25T19:46:00.616 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.614+0000 7f5b272f9640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b200ab430 0x7f5b200b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:00.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.614+0000 7f5b272f9640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b2014d490 0x7f5b2006f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:00.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.614+0000 7f5b272f9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b201d2fb0 0x7f5b20134220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:00.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.614+0000 7f5b272f9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2013b210 con 0x7f5b201d2fb0 2024-04-25T19:46:00.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.614+0000 7f5b272f9640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5b201482a0 con 0x7f5b200ab430 2024-04-25T19:46:00.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.614+0000 7f5b272f9640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f5b2013b390 con 0x7f5b2014d490 2024-04-25T19:46:00.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.614+0000 7f5b25896640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b201d2fb0 0x7f5b20134220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:00.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.614+0000 7f5b24894640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b2014d490 0x7f5b2006f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:00.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.614+0000 7f5b25896640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b201d2fb0 0x7f5b20134220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51154/0 (socket says 172.21.15.29:51154) 2024-04-25T19:46:00.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.614+0000 7f5b25896640 1 -- 172.21.15.29:0/2260420438 learned_addr learned my addr 172.21.15.29:0/2260420438 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:46:00.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.614+0000 7f5b25095640 1 --2- 172.21.15.29:0/2260420438 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b200ab430 0x7f5b200b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:00.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.614+0000 7f5b25896640 1 -- 172.21.15.29:0/2260420438 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b2014d490 msgr2=0x7f5b2006f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:00.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.614+0000 7f5b25896640 1 --2- 172.21.15.29:0/2260420438 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b2014d490 0x7f5b2006f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:00.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.614+0000 7f5b25896640 1 -- 172.21.15.29:0/2260420438 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b200ab430 msgr2=0x7f5b200b3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:00.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.614+0000 7f5b25896640 1 --2- 172.21.15.29:0/2260420438 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b200ab430 0x7f5b200b3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:00.617 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.614+0000 7f5b25896640 1 -- 172.21.15.29:0/2260420438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b200b0160 con 0x7f5b201d2fb0 2024-04-25T19:46:00.618 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.614+0000 7f5b24894640 1 --2- 172.21.15.29:0/2260420438 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b2014d490 0x7f5b2006f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:46:00.618 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.615+0000 7f5b25896640 1 --2- 172.21.15.29:0/2260420438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b201d2fb0 0x7f5b20134220 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800efc0 tx=0x7f5b1800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:00.618 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.615+0000 7f5b15ffb640 1 -- 172.21.15.29:0/2260420438 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b18018020 con 0x7f5b201d2fb0 2024-04-25T19:46:00.618 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.615+0000 7f5b15ffb640 1 -- 172.21.15.29:0/2260420438 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b18014480 con 0x7f5b201d2fb0 2024-04-25T19:46:00.618 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.615+0000 7f5b272f9640 1 -- 172.21.15.29:0/2260420438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b200b3b70 con 0x7f5b201d2fb0 2024-04-25T19:46:00.618 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.615+0000 7f5b15ffb640 1 -- 172.21.15.29:0/2260420438 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b18009e60 con 0x7f5b201d2fb0 2024-04-25T19:46:00.618 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.615+0000 7f5b272f9640 1 -- 172.21.15.29:0/2260420438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b20071250 con 0x7f5b201d2fb0 2024-04-25T19:46:00.618 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.616+0000 7f5b272f9640 1 -- 172.21.15.29:0/2260420438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b200692d0 con 0x7f5b201d2fb0 2024-04-25T19:46:00.619 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.618+0000 7f5b15ffb640 1 -- 172.21.15.29:0/2260420438 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) ==== 144961+0+0 (secure 0 0 0) 0x7f5b18005d10 con 0x7f5b201d2fb0 2024-04-25T19:46:00.619 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.618+0000 7f5b15ffb640 20 mgrc handle_mgr_map mgrmap(e 12) 2024-04-25T19:46:00.620 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.618+0000 7f5b15ffb640 4 mgrc handle_mgr_map Got map version 12 2024-04-25T19:46:00.620 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.618+0000 7f5b15ffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] 2024-04-25T19:46:00.620 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.618+0000 7f5b15ffb640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] 2024-04-25T19:46:00.620 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.618+0000 7f5b15ffb640 1 --2- 172.21.15.29:0/2260420438 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f5b040ac650 0x7f5b040aeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:00.620 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.618+0000 7f5b25095640 1 -- 172.21.15.29:0/2260420438 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f5b040ac650 msgr2=0x7f5b040aeb60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6824/1500823528 2024-04-25T19:46:00.620 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.618+0000 7f5b15ffb640 1 -- 172.21.15.29:0/2260420438 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5566+0+0 (secure 0 0 0) 0x7f5b180daeb0 con 0x7f5b201d2fb0 2024-04-25T19:46:00.620 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.619+0000 7f5b25095640 1 --2- 172.21.15.29:0/2260420438 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f5b040ac650 0x7f5b040aeb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T19:46:00.627 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.626+0000 7f5b15ffb640 1 -- 172.21.15.29:0/2260420438 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5b18109b80 con 0x7f5b201d2fb0 2024-04-25T19:46:00.705 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:46:00.704+0000 7f33fe212040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:46:00.709 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:46:00.708+0000 7f846c637040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:46:00.709 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:00.708+0000 7f4e1e83c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:46:00.813 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.812+0000 7f5b272f9640 1 -- 172.21.15.29:0/2260420438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "by-host-smithi029", "pool2": "by-host-smithi029", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5b2006cf50 con 0x7f5b201d2fb0 2024-04-25T19:46:00.821 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.820+0000 7f5b25095640 1 -- 172.21.15.29:0/2260420438 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f5b040ac650 msgr2=0x7f5b040aeb60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6824/1500823528 2024-04-25T19:46:00.821 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:00.820+0000 7f5b25095640 1 --2- 172.21.15.29:0/2260420438 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f5b040ac650 0x7f5b040aeb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T19:46:01.030 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.029+0000 7f5b15ffb640 1 -- 172.21.15.29:0/2260420438 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "by-host-smithi029", "pool2": "by-host-smithi029", "yes_i_really_really_mean_it": true}]=0 pool 'by-host-smithi029' removed v29) ==== 187+0+0 (secure 0 0 0) 0x7f5b18022020 con 0x7f5b201d2fb0 2024-04-25T19:46:01.039 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:01.038+0000 7f4e1e83c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T19:46:01.039 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:46:01.038+0000 7f846c637040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T19:46:01.048 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:46:01.047+0000 7f33fe212040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T19:46:01.177 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.175+0000 7f5b272f9640 1 -- 172.21.15.29:0/2260420438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "by-host-smithi029", "pool2": "by-host-smithi029", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5b20070080 con 0x7f5b201d2fb0 2024-04-25T19:46:01.177 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.175+0000 7f5b15ffb640 1 -- 172.21.15.29:0/2260420438 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "by-host-smithi029", "pool2": "by-host-smithi029", "yes_i_really_really_mean_it": true}]=0 pool 'by-host-smithi029' does not exist v29) ==== 194+0+0 (secure 0 0 0) 0x7f5b18010040 con 0x7f5b201d2fb0 2024-04-25T19:46:01.177 INFO:tasks.workunit.client.0.smithi029.stderr:pool 'by-host-smithi029' does not exist 2024-04-25T19:46:01.179 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.177+0000 7f5b272f9640 10 mgrc shutdown 2024-04-25T19:46:01.179 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.177+0000 7f5b272f9640 1 -- 172.21.15.29:0/2260420438 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f5b040ac650 msgr2=0x7f5b040aeb60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T19:46:01.179 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.177+0000 7f5b272f9640 1 --2- 172.21.15.29:0/2260420438 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f5b040ac650 0x7f5b040aeb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:01.179 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.177+0000 7f5b272f9640 1 -- 172.21.15.29:0/2260420438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b201d2fb0 msgr2=0x7f5b20134220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:01.179 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.177+0000 7f5b272f9640 1 --2- 172.21.15.29:0/2260420438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b201d2fb0 0x7f5b20134220 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800efc0 tx=0x7f5b1800c490 comp rx=0 tx=0).stop 2024-04-25T19:46:01.179 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.178+0000 7f5b272f9640 1 -- 172.21.15.29:0/2260420438 shutdown_connections 2024-04-25T19:46:01.180 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.178+0000 7f5b272f9640 1 --2- 172.21.15.29:0/2260420438 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f5b040ac650 0x7f5b040aeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:01.180 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.178+0000 7f5b272f9640 1 --2- 172.21.15.29:0/2260420438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b201d2fb0 0x7f5b20134220 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:01.180 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.178+0000 7f5b272f9640 1 --2- 172.21.15.29:0/2260420438 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b2014d490 0x7f5b2006f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:01.180 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.178+0000 7f5b272f9640 1 --2- 172.21.15.29:0/2260420438 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b200ab430 0x7f5b200b3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:01.180 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.178+0000 7f5b272f9640 1 -- 172.21.15.29:0/2260420438 >> 172.21.15.29:0/2260420438 conn(0x7f5b200b6c60 msgr2=0x7f5b20135480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:01.180 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.178+0000 7f5b272f9640 1 -- 172.21.15.29:0/2260420438 shutdown_connections 2024-04-25T19:46:01.180 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.178+0000 7f5b272f9640 1 -- 172.21.15.29:0/2260420438 wait complete. 2024-04-25T19:46:01.188 INFO:tasks.workunit.client.0.smithi029.stderr:+ for p in `ceph osd pool ls | grep '^by-host-'` 2024-04-25T19:46:01.188 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph osd pool rm by-host-smithi102 by-host-smithi102 --yes-i-really-really-mean-it 2024-04-25T19:46:01.283 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.282+0000 7f73f853c640 1 Processor -- start 2024-04-25T19:46:01.283 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.282+0000 7f73f853c640 1 -- start start 2024-04-25T19:46:01.284 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.283+0000 7f73f853c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f73f00a2b20 0x7f73f00a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:01.285 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.283+0000 7f73f853c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f73f0147200 0x7f73f0147610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:01.285 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.283+0000 7f73f853c640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73f00a7de0 con 0x7f73f00b5b90 2024-04-25T19:46:01.285 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.283+0000 7f73f853c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f73f00a5a20 con 0x7f73f00a2b20 2024-04-25T19:46:01.285 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.283+0000 7f73f853c640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f73f00a3520 con 0x7f73f0147200 2024-04-25T19:46:01.285 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.283+0000 7f73f5ad7640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f73f0147200 0x7f73f0147610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:01.285 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.283+0000 7f73f62d8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f73f00a2b20 0x7f73f00a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:01.285 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.283+0000 7f73f5ad7640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f73f0147200 0x7f73f0147610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:49914/0 (socket says 172.21.15.29:49914) 2024-04-25T19:46:01.285 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.283+0000 7f73f5ad7640 1 -- 172.21.15.29:0/803063656 learned_addr learned my addr 172.21.15.29:0/803063656 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:46:01.285 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.283+0000 7f73f5ad7640 1 -- 172.21.15.29:0/803063656 >> v2:172.21.15.29:3300/0 conn(0x7f73f00a2b20 msgr2=0x7f73f00a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:46:01.285 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.283+0000 7f73f5ad7640 1 --2- 172.21.15.29:0/803063656 >> v2:172.21.15.29:3300/0 conn(0x7f73f00a2b20 0x7f73f00a1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:01.285 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.283+0000 7f73f5ad7640 1 -- 172.21.15.29:0/803063656 >> v1:172.21.15.102:6789/0 conn(0x7f73f00b5b90 legacy=0x7f73f00b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:01.285 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.283+0000 7f73f5ad7640 1 -- 172.21.15.29:0/803063656 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f0147c40 con 0x7f73f0147200 2024-04-25T19:46:01.285 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.283+0000 7f73f62d8640 1 --2- 172.21.15.29:0/803063656 >> v2:172.21.15.29:3300/0 conn(0x7f73f00a2b20 0x7f73f00a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:46:01.287 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.284+0000 7f73f5ad7640 1 --2- 172.21.15.29:0/803063656 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f73f0147200 0x7f73f0147610 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f73e0009870 tx=0x7f73e002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d1182a9fcee7b43 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:01.287 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.284+0000 7f73f52d6640 1 -- 172.21.15.29:0/803063656 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e0035020 con 0x7f73f0147200 2024-04-25T19:46:01.287 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.284+0000 7f73f52d6640 1 -- 172.21.15.29:0/803063656 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73e0031640 con 0x7f73f0147200 2024-04-25T19:46:01.287 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.284+0000 7f73f52d6640 1 -- 172.21.15.29:0/803063656 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e002d040 con 0x7f73f0147200 2024-04-25T19:46:01.287 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.285+0000 7f73f853c640 1 -- 172.21.15.29:0/803063656 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f73f0147200 msgr2=0x7f73f0147610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:01.287 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.285+0000 7f73f853c640 1 --2- 172.21.15.29:0/803063656 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f73f0147200 0x7f73f0147610 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f73e0009870 tx=0x7f73e002b8f0 comp rx=0 tx=0).stop 2024-04-25T19:46:01.287 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.285+0000 7f73f853c640 1 -- 172.21.15.29:0/803063656 shutdown_connections 2024-04-25T19:46:01.287 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.285+0000 7f73f853c640 1 --2- 172.21.15.29:0/803063656 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f73f0147200 0x7f73f0147610 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:01.287 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.285+0000 7f73f853c640 1 --2- 172.21.15.29:0/803063656 >> v2:172.21.15.29:3300/0 conn(0x7f73f00a2b20 0x7f73f00a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:01.287 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.285+0000 7f73f853c640 1 -- 172.21.15.29:0/803063656 >> 172.21.15.29:0/803063656 conn(0x7f73f006df60 msgr2=0x7f73f006e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:01.287 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.285+0000 7f73f853c640 1 -- 172.21.15.29:0/803063656 shutdown_connections 2024-04-25T19:46:01.287 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.285+0000 7f73f853c640 1 -- 172.21.15.29:0/803063656 wait complete. 2024-04-25T19:46:01.288 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.287+0000 7f73f853c640 1 Processor -- start 2024-04-25T19:46:01.288 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.287+0000 7f73f853c640 1 -- start start 2024-04-25T19:46:01.290 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.288+0000 7f73f853c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73f00a2b20 0x7f73f00b9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:01.290 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.288+0000 7f73f853c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f00b5b90 0x7f73f0142ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:01.290 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.288+0000 7f73f853c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f73f0147200 0x7f73f0073d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:01.290 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.288+0000 7f73f853c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f73f00a2f30 con 0x7f73f00b5b90 2024-04-25T19:46:01.290 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.288+0000 7f73f853c640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f73f0147f00 con 0x7f73f00a2b20 2024-04-25T19:46:01.290 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.288+0000 7f73f853c640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f73f0143520 con 0x7f73f0147200 2024-04-25T19:46:01.290 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.288+0000 7f73f5ad7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f00b5b90 0x7f73f0142ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:01.290 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.288+0000 7f73f5ad7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f00b5b90 0x7f73f0142ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51172/0 (socket says 172.21.15.29:51172) 2024-04-25T19:46:01.290 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.288+0000 7f73f6ad9640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f73f0147200 0x7f73f0073d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:01.291 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.288+0000 7f73f5ad7640 1 -- 172.21.15.29:0/2675257978 learned_addr learned my addr 172.21.15.29:0/2675257978 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:46:01.291 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.288+0000 7f73f62d8640 1 --2- 172.21.15.29:0/2675257978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73f00a2b20 0x7f73f00b9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:01.291 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.288+0000 7f73f5ad7640 1 -- 172.21.15.29:0/2675257978 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f73f0147200 msgr2=0x7f73f0073d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:01.291 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.288+0000 7f73f5ad7640 1 --2- 172.21.15.29:0/2675257978 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f73f0147200 0x7f73f0073d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:01.291 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.288+0000 7f73f5ad7640 1 -- 172.21.15.29:0/2675257978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73f00a2b20 msgr2=0x7f73f00b9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:01.291 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.288+0000 7f73f5ad7640 1 --2- 172.21.15.29:0/2675257978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73f00a2b20 0x7f73f00b9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:01.291 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.288+0000 7f73f5ad7640 1 -- 172.21.15.29:0/2675257978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f0150350 con 0x7f73f00b5b90 2024-04-25T19:46:01.291 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.288+0000 7f73f5ad7640 1 --2- 172.21.15.29:0/2675257978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f00b5b90 0x7f73f0142ee0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f73e002b930 tx=0x7f73e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:01.291 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.288+0000 7f73e6ffd640 1 -- 172.21.15.29:0/2675257978 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e0040070 con 0x7f73f00b5b90 2024-04-25T19:46:01.291 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.289+0000 7f73e6ffd640 1 -- 172.21.15.29:0/2675257978 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73e003e070 con 0x7f73f00b5b90 2024-04-25T19:46:01.291 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.289+0000 7f73f853c640 1 -- 172.21.15.29:0/2675257978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73f0144560 con 0x7f73f00b5b90 2024-04-25T19:46:01.291 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.289+0000 7f73e6ffd640 1 -- 172.21.15.29:0/2675257978 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e0035040 con 0x7f73f00b5b90 2024-04-25T19:46:01.291 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.289+0000 7f73f853c640 1 -- 172.21.15.29:0/2675257978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73f01df420 con 0x7f73f00b5b90 2024-04-25T19:46:01.292 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.289+0000 7f73f853c640 1 -- 172.21.15.29:0/2675257978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73f00692d0 con 0x7f73f00b5b90 2024-04-25T19:46:01.292 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.291+0000 7f73e6ffd640 1 -- 172.21.15.29:0/2675257978 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) ==== 144961+0+0 (secure 0 0 0) 0x7f73e0038020 con 0x7f73f00b5b90 2024-04-25T19:46:01.292 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.291+0000 7f73e6ffd640 20 mgrc handle_mgr_map mgrmap(e 12) 2024-04-25T19:46:01.293 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.292+0000 7f73e6ffd640 4 mgrc handle_mgr_map Got map version 12 2024-04-25T19:46:01.293 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.292+0000 7f73e6ffd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] 2024-04-25T19:46:01.293 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.292+0000 7f73e6ffd640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] 2024-04-25T19:46:01.293 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.292+0000 7f73e6ffd640 1 --2- 172.21.15.29:0/2675257978 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f73d40ac680 0x7f73d40aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:01.293 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.292+0000 7f73f62d8640 1 -- 172.21.15.29:0/2675257978 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f73d40ac680 msgr2=0x7f73d40aeb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6824/1500823528 2024-04-25T19:46:01.293 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.292+0000 7f73e6ffd640 1 -- 172.21.15.29:0/2675257978 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5210+0+0 (secure 0 0 0) 0x7f73e00fb720 con 0x7f73f00b5b90 2024-04-25T19:46:01.293 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.292+0000 7f73f62d8640 1 --2- 172.21.15.29:0/2675257978 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f73d40ac680 0x7f73d40aeb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T19:46:01.295 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:46:01.294+0000 7f846c637040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T19:46:01.295 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:01.295+0000 7f4e1e83c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T19:46:01.301 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.300+0000 7f73e6ffd640 1 -- 172.21.15.29:0/2675257978 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f73e0053020 con 0x7f73f00b5b90 2024-04-25T19:46:01.302 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:46:01.301+0000 7f33fe212040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T19:46:01.347 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:01.346+0000 7f4e1e83c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T19:46:01.347 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:46:01.346+0000 7f846c637040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T19:46:01.355 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:46:01.354+0000 7f33fe212040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T19:46:01.465 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:01.465+0000 7f4e1e83c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T19:46:01.466 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:46:01.465+0000 7f846c637040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T19:46:01.475 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:46:01.474+0000 7f33fe212040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T19:46:01.491 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.490+0000 7f73f853c640 1 -- 172.21.15.29:0/2675257978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "by-host-smithi102", "pool2": "by-host-smithi102", "yes_i_really_really_mean_it": true} v 0) -- 0x7f73f0145cf0 con 0x7f73f00b5b90 2024-04-25T19:46:01.493 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.493+0000 7f73f62d8640 1 -- 172.21.15.29:0/2675257978 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f73d40ac680 msgr2=0x7f73d40aeb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6824/1500823528 2024-04-25T19:46:01.494 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.493+0000 7f73f62d8640 1 --2- 172.21.15.29:0/2675257978 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f73d40ac680 0x7f73d40aeb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T19:46:01.737 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:01.737+0000 7f4e1e83c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T19:46:01.738 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:46:01.737+0000 7f846c637040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T19:46:01.758 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:46:01.757+0000 7f33fe212040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T19:46:01.781 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:01.781+0000 7f4e1e83c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T19:46:01.781 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:46:01.781+0000 7f846c637040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T19:46:01.804 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:46:01.803+0000 7f33fe212040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T19:46:01.895 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.894+0000 7f73f62d8640 1 -- 172.21.15.29:0/2675257978 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f73d40ac680 msgr2=0x7f73d40aeb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6824/1500823528 2024-04-25T19:46:01.895 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:01.894+0000 7f73f62d8640 1 --2- 172.21.15.29:0/2675257978 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f73d40ac680 0x7f73d40aeb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T19:46:01.928 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:01.927+0000 7f4e1e83c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T19:46:01.928 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:46:01.928+0000 7f846c637040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T19:46:01.955 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:46:01.954+0000 7f33fe212040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T19:46:02.032 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.031+0000 7f73e6ffd640 1 -- 172.21.15.29:0/2675257978 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "by-host-smithi102", "pool2": "by-host-smithi102", "yes_i_really_really_mean_it": true}]=0 pool 'by-host-smithi102' removed v30) ==== 187+0+0 (secure 0 0 0) 0x7f73e00c5f10 con 0x7f73f00b5b90 2024-04-25T19:46:02.076 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:46:02.076+0000 7f846c637040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T19:46:02.077 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:02.077+0000 7f4e1e83c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T19:46:02.099 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:46:02.098+0000 7f33fe212040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T19:46:02.130 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:02.129+0000 7f4e1e83c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T19:46:02.130 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:46:02.130+0000 7f846c637040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T19:46:02.144 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.143+0000 7f73f853c640 1 -- 172.21.15.29:0/2675257978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "by-host-smithi102", "pool2": "by-host-smithi102", "yes_i_really_really_mean_it": true} v 0) -- 0x7f73f00a4aa0 con 0x7f73f00b5b90 2024-04-25T19:46:02.145 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.144+0000 7f73e6ffd640 1 -- 172.21.15.29:0/2675257978 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "by-host-smithi102", "pool2": "by-host-smithi102", "yes_i_really_really_mean_it": true}]=0 pool 'by-host-smithi102' does not exist v30) ==== 194+0+0 (secure 0 0 0) 0x7f73e00c5810 con 0x7f73f00b5b90 2024-04-25T19:46:02.145 INFO:tasks.workunit.client.0.smithi029.stderr:pool 'by-host-smithi102' does not exist 2024-04-25T19:46:02.147 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.146+0000 7f73f853c640 10 mgrc shutdown 2024-04-25T19:46:02.147 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.146+0000 7f73f853c640 1 -- 172.21.15.29:0/2675257978 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f73d40ac680 msgr2=0x7f73d40aeb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T19:46:02.147 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.146+0000 7f73f853c640 1 --2- 172.21.15.29:0/2675257978 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f73d40ac680 0x7f73d40aeb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.148 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.146+0000 7f73f853c640 1 -- 172.21.15.29:0/2675257978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f00b5b90 msgr2=0x7f73f0142ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:02.148 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.146+0000 7f73f853c640 1 --2- 172.21.15.29:0/2675257978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f00b5b90 0x7f73f0142ee0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f73e002b930 tx=0x7f73e00046c0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.148 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.146+0000 7f73f853c640 1 -- 172.21.15.29:0/2675257978 shutdown_connections 2024-04-25T19:46:02.148 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.146+0000 7f73f853c640 1 --2- 172.21.15.29:0/2675257978 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f73d40ac680 0x7f73d40aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.148 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.146+0000 7f73f853c640 1 --2- 172.21.15.29:0/2675257978 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f73f0147200 0x7f73f0073d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.148 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.146+0000 7f73f853c640 1 --2- 172.21.15.29:0/2675257978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f00b5b90 0x7f73f0142ee0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.148 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.146+0000 7f73f853c640 1 --2- 172.21.15.29:0/2675257978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73f00a2b20 0x7f73f00b9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.148 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.146+0000 7f73f853c640 1 -- 172.21.15.29:0/2675257978 >> 172.21.15.29:0/2675257978 conn(0x7f73f006df60 msgr2=0x7f73f00a5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:02.148 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.146+0000 7f73f853c640 1 -- 172.21.15.29:0/2675257978 shutdown_connections 2024-04-25T19:46:02.148 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.146+0000 7f73f853c640 1 -- 172.21.15.29:0/2675257978 wait complete. 2024-04-25T19:46:02.151 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:46:02.150+0000 7f33fe212040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T19:46:02.157 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph config rm mgr mgr/localpool/subtree 2024-04-25T19:46:02.198 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:02.198+0000 7f4e1e83c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T19:46:02.199 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:46:02.198+0000 7f846c637040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T19:46:02.210 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:46:02.209+0000 7f33fe212040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T19:46:02.244 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:02.243+0000 7f4e1e83c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T19:46:02.244 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:46:02.243+0000 7f846c637040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T19:46:02.249 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.248+0000 7f79e1d30640 1 Processor -- start 2024-04-25T19:46:02.250 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.248+0000 7f79e1d30640 1 -- start start 2024-04-25T19:46:02.252 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.249+0000 7f79e1d30640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f79dc0b5f20 0x7f79dc0730e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:02.252 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.249+0000 7f79e1d30640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f79dc0aa6d0 0x7f79dc1433e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:02.252 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.249+0000 7f79e1d30640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f79dc143920 0x7f79dc071b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:02.252 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.249+0000 7f79e1d30640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f79dc0aae00 con 0x7f79dc143920 2024-04-25T19:46:02.252 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.249+0000 7f79e1d30640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f79dc0ab0c0 con 0x7f79dc0b5f20 2024-04-25T19:46:02.252 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.249+0000 7f79e1d30640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f79dc0aaae0 con 0x7f79dc0aa6d0 2024-04-25T19:46:02.252 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.250+0000 7f79dbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f79dc143920 0x7f79dc071b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:02.252 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.250+0000 7f79daffd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f79dc0aa6d0 0x7f79dc1433e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:02.252 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.250+0000 7f79dbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f79dc143920 0x7f79dc071b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51180/0 (socket says 172.21.15.29:51180) 2024-04-25T19:46:02.252 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.250+0000 7f79dbfff640 1 -- 172.21.15.29:0/3914321668 learned_addr learned my addr 172.21.15.29:0/3914321668 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:46:02.252 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.250+0000 7f79db7fe640 1 --2- 172.21.15.29:0/3914321668 >> v2:172.21.15.102:3300/0 conn(0x7f79dc0b5f20 0x7f79dc0730e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:02.253 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.250+0000 7f79dbfff640 1 -- 172.21.15.29:0/3914321668 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f79dc0aa6d0 msgr2=0x7f79dc1433e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:46:02.253 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.250+0000 7f79dbfff640 1 --2- 172.21.15.29:0/3914321668 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f79dc0aa6d0 0x7f79dc1433e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.253 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.250+0000 7f79dbfff640 1 -- 172.21.15.29:0/3914321668 >> v2:172.21.15.102:3300/0 conn(0x7f79dc0b5f20 msgr2=0x7f79dc0730e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:46:02.253 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.250+0000 7f79dbfff640 1 --2- 172.21.15.29:0/3914321668 >> v2:172.21.15.102:3300/0 conn(0x7f79dc0b5f20 0x7f79dc0730e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.253 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.250+0000 7f79dbfff640 1 -- 172.21.15.29:0/3914321668 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79dc0b5130 con 0x7f79dc143920 2024-04-25T19:46:02.253 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.250+0000 7f79db7fe640 1 --2- 172.21.15.29:0/3914321668 >> v2:172.21.15.102:3300/0 conn(0x7f79dc0b5f20 0x7f79dc0730e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:46:02.253 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.250+0000 7f79daffd640 1 --2- 172.21.15.29:0/3914321668 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f79dc0aa6d0 0x7f79dc1433e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:46:02.253 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.251+0000 7f79dbfff640 1 --2- 172.21.15.29:0/3914321668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79dc143920 0x7f79dc071b00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f79dc0b6ad0 tx=0x7f79d402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e57c0251cb0c1d96 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:02.254 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.251+0000 7f79da7fc640 1 -- 172.21.15.29:0/3914321668 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d402cb00 con 0x7f79dc143920 2024-04-25T19:46:02.254 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.251+0000 7f79da7fc640 1 -- 172.21.15.29:0/3914321668 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79d402cca0 con 0x7f79dc143920 2024-04-25T19:46:02.254 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.251+0000 7f79da7fc640 1 -- 172.21.15.29:0/3914321668 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d4034610 con 0x7f79dc143920 2024-04-25T19:46:02.254 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.252+0000 7f79e1d30640 1 -- 172.21.15.29:0/3914321668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79dc143920 msgr2=0x7f79dc071b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:02.254 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.252+0000 7f79e1d30640 1 --2- 172.21.15.29:0/3914321668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79dc143920 0x7f79dc071b00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f79dc0b6ad0 tx=0x7f79d402bf60 comp rx=0 tx=0).stop 2024-04-25T19:46:02.255 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.252+0000 7f79e1d30640 1 -- 172.21.15.29:0/3914321668 shutdown_connections 2024-04-25T19:46:02.255 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.252+0000 7f79e1d30640 1 --2- 172.21.15.29:0/3914321668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79dc143920 0x7f79dc071b00 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.255 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.252+0000 7f79e1d30640 1 --2- 172.21.15.29:0/3914321668 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f79dc0aa6d0 0x7f79dc1433e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.255 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.252+0000 7f79e1d30640 1 --2- 172.21.15.29:0/3914321668 >> v2:172.21.15.102:3300/0 conn(0x7f79dc0b5f20 0x7f79dc0730e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.255 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.252+0000 7f79e1d30640 1 -- 172.21.15.29:0/3914321668 >> 172.21.15.29:0/3914321668 conn(0x7f79dc06df60 msgr2=0x7f79dc06e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:02.255 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.252+0000 7f79e1d30640 1 -- 172.21.15.29:0/3914321668 shutdown_connections 2024-04-25T19:46:02.255 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.252+0000 7f79e1d30640 1 -- 172.21.15.29:0/3914321668 wait complete. 2024-04-25T19:46:02.257 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.254+0000 7f79e1d30640 1 Processor -- start 2024-04-25T19:46:02.258 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.254+0000 7f79e1d30640 1 -- start start 2024-04-25T19:46:02.258 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.255+0000 7f79e1d30640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79dc0aa6d0 0x7f79dc13df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:02.258 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.255+0000 7f79e1d30640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79dc0b5f20 0x7f79dc0ad680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:02.258 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.255+0000 7f79e1d30640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f79dc0b8d10 0x7f79dc0ae820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:02.258 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.255+0000 7f79e1d30640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc0abd00 con 0x7f79dc0b5f20 2024-04-25T19:46:02.258 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.255+0000 7f79e1d30640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc0abb80 con 0x7f79dc0aa6d0 2024-04-25T19:46:02.258 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.255+0000 7f79e1d30640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f79dc072040 con 0x7f79dc0b8d10 2024-04-25T19:46:02.258 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.255+0000 7f79daffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79dc0b5f20 0x7f79dc0ad680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:02.258 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.255+0000 7f79dbfff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f79dc0b8d10 0x7f79dc0ae820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:02.258 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.255+0000 7f79dbfff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f79dc0b8d10 0x7f79dc0ae820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:49946/0 (socket says 172.21.15.29:49946) 2024-04-25T19:46:02.258 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.255+0000 7f79daffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79dc0b5f20 0x7f79dc0ad680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51190/0 (socket says 172.21.15.29:51190) 2024-04-25T19:46:02.258 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.255+0000 7f79dbfff640 1 -- 172.21.15.29:0/1446509693 learned_addr learned my addr 172.21.15.29:0/1446509693 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:46:02.258 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.255+0000 7f79db7fe640 1 --2- 172.21.15.29:0/1446509693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79dc0aa6d0 0x7f79dc13df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:02.258 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.255+0000 7f79dbfff640 1 -- 172.21.15.29:0/1446509693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79dc0aa6d0 msgr2=0x7f79dc13df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:02.259 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.255+0000 7f79dbfff640 1 --2- 172.21.15.29:0/1446509693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79dc0aa6d0 0x7f79dc13df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.259 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.255+0000 7f79dbfff640 1 -- 172.21.15.29:0/1446509693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79dc0b5f20 msgr2=0x7f79dc0ad680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:02.259 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.255+0000 7f79dbfff640 1 --2- 172.21.15.29:0/1446509693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79dc0b5f20 0x7f79dc0ad680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.259 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.255+0000 7f79dbfff640 1 -- 172.21.15.29:0/1446509693 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79dc1d84b0 con 0x7f79dc0b8d10 2024-04-25T19:46:02.259 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.255+0000 7f79daffd640 1 --2- 172.21.15.29:0/1446509693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79dc0b5f20 0x7f79dc0ad680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:46:02.259 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.256+0000 7f79dbfff640 1 --2- 172.21.15.29:0/1446509693 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f79dc0b8d10 0x7f79dc0ae820 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009f40 tx=0x7f79d4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:02.259 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.256+0000 7f79d3fff640 1 -- 172.21.15.29:0/1446509693 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d4043020 con 0x7f79dc0b8d10 2024-04-25T19:46:02.259 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.256+0000 7f79d3fff640 1 -- 172.21.15.29:0/1446509693 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79d402ce50 con 0x7f79dc0b8d10 2024-04-25T19:46:02.259 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.256+0000 7f79e1d30640 1 -- 172.21.15.29:0/1446509693 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79dc0aed60 con 0x7f79dc0b8d10 2024-04-25T19:46:02.259 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.256+0000 7f79d3fff640 1 -- 172.21.15.29:0/1446509693 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d4009510 con 0x7f79dc0b8d10 2024-04-25T19:46:02.259 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.256+0000 7f79e1d30640 1 -- 172.21.15.29:0/1446509693 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f79dc1d87e0 con 0x7f79dc0b8d10 2024-04-25T19:46:02.260 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:46:02.256+0000 7f33fe212040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T19:46:02.261 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.257+0000 7f79e1d30640 1 -- 172.21.15.29:0/1446509693 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79dc0692d0 con 0x7f79dc0b8d10 2024-04-25T19:46:02.261 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.258+0000 7f79d3fff640 1 -- 172.21.15.29:0/1446509693 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 12) ==== 144961+0+0 (secure 0 0 0) 0x7f79d4046020 con 0x7f79dc0b8d10 2024-04-25T19:46:02.261 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.258+0000 7f79d3fff640 20 mgrc handle_mgr_map mgrmap(e 12) 2024-04-25T19:46:02.261 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.259+0000 7f79d3fff640 4 mgrc handle_mgr_map Got map version 12 2024-04-25T19:46:02.261 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.259+0000 7f79d3fff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] 2024-04-25T19:46:02.261 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.259+0000 7f79d3fff640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] 2024-04-25T19:46:02.261 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.259+0000 7f79d3fff640 1 --2- 172.21.15.29:0/1446509693 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f79bc0ac650 0x7f79bc0aeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:02.261 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.259+0000 7f79db7fe640 1 -- 172.21.15.29:0/1446509693 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f79bc0ac650 msgr2=0x7f79bc0aeb60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6824/1500823528 2024-04-25T19:46:02.261 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.259+0000 7f79d3fff640 1 -- 172.21.15.29:0/1446509693 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 4854+0+0 (secure 0 0 0) 0x7f79d40fbe80 con 0x7f79dc0b8d10 2024-04-25T19:46:02.261 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.259+0000 7f79db7fe640 1 --2- 172.21.15.29:0/1446509693 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f79bc0ac650 0x7f79bc0aeb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T19:46:02.264 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.263+0000 7f79d3fff640 1 -- 172.21.15.29:0/1446509693 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f79d404d020 con 0x7f79dc0b8d10 2024-04-25T19:46:02.461 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.460+0000 7f79db7fe640 1 -- 172.21.15.29:0/1446509693 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f79bc0ac650 msgr2=0x7f79bc0aeb60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6824/1500823528 2024-04-25T19:46:02.461 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.460+0000 7f79db7fe640 1 --2- 172.21.15.29:0/1446509693 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f79bc0ac650 0x7f79bc0aeb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T19:46:02.482 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.480+0000 7f79e1d30640 1 -- 172.21.15.29:0/1446509693 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/localpool/subtree"} v 0) -- 0x7f79dc072ec0 con 0x7f79dc0b8d10 2024-04-25T19:46:02.488 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.487+0000 7f79d3fff640 1 -- 172.21.15.29:0/1446509693 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/localpool/subtree"}]=0 v8) ==== 104+0+0 (secure 0 0 0) 0x7f79d40c67d0 con 0x7f79dc0b8d10 2024-04-25T19:46:02.597 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.596+0000 7f79e1d30640 1 -- 172.21.15.29:0/1446509693 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/localpool/subtree"} v 0) -- 0x7f79dc0008d0 con 0x7f79dc0b8d10 2024-04-25T19:46:02.598 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.597+0000 7f79d3fff640 1 -- 172.21.15.29:0/1446509693 <== mon.2 v2:172.21.15.29:3301/0 8 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/localpool/subtree"}]=0 v8) ==== 104+0+0 (secure 0 0 0) 0x7f79d40c60d0 con 0x7f79dc0b8d10 2024-04-25T19:46:02.601 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.599+0000 7f79e1d30640 10 mgrc shutdown 2024-04-25T19:46:02.601 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.599+0000 7f79e1d30640 1 -- 172.21.15.29:0/1446509693 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f79bc0ac650 msgr2=0x7f79bc0aeb60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T19:46:02.601 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.599+0000 7f79e1d30640 1 --2- 172.21.15.29:0/1446509693 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f79bc0ac650 0x7f79bc0aeb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.601 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.599+0000 7f79e1d30640 1 -- 172.21.15.29:0/1446509693 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f79dc0b8d10 msgr2=0x7f79dc0ae820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:02.601 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.599+0000 7f79e1d30640 1 --2- 172.21.15.29:0/1446509693 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f79dc0b8d10 0x7f79dc0ae820 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009f40 tx=0x7f79d4004a40 comp rx=0 tx=0).stop 2024-04-25T19:46:02.601 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.599+0000 7f79e1d30640 1 -- 172.21.15.29:0/1446509693 shutdown_connections 2024-04-25T19:46:02.601 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.599+0000 7f79e1d30640 1 --2- 172.21.15.29:0/1446509693 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f79bc0ac650 0x7f79bc0aeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.601 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.599+0000 7f79e1d30640 1 --2- 172.21.15.29:0/1446509693 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f79dc0b8d10 0x7f79dc0ae820 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.601 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.599+0000 7f79e1d30640 1 --2- 172.21.15.29:0/1446509693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79dc0b5f20 0x7f79dc0ad680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.601 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.599+0000 7f79e1d30640 1 --2- 172.21.15.29:0/1446509693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79dc0aa6d0 0x7f79dc13df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.601 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.599+0000 7f79e1d30640 1 -- 172.21.15.29:0/1446509693 >> 172.21.15.29:0/1446509693 conn(0x7f79dc06df60 msgr2=0x7f79dc0724e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:02.601 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.599+0000 7f79e1d30640 1 -- 172.21.15.29:0/1446509693 shutdown_connections 2024-04-25T19:46:02.602 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.599+0000 7f79e1d30640 1 -- 172.21.15.29:0/1446509693 wait complete. 2024-04-25T19:46:02.610 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph config rm mgr mgr/localpool/failure_domain 2024-04-25T19:46:02.656 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:02.656+0000 7f4e1e83c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T19:46:02.659 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:46:02.659+0000 7f846c637040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T19:46:02.701 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:46:02.700+0000 7f33fe212040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T19:46:02.702 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.701+0000 7f64ccb2a640 1 Processor -- start 2024-04-25T19:46:02.702 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.701+0000 7f64ccb2a640 1 -- start start 2024-04-25T19:46:02.704 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.702+0000 7f64ccb2a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f64c806ce50 0x7f64c80a5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:02.704 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.702+0000 7f64ccb2a640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f64c814af10 0x7f64c8151840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:02.704 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.702+0000 7f64ccb2a640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64c80a6850 con 0x7f64c80a6440 2024-04-25T19:46:02.704 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.702+0000 7f64ccb2a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f64c80a6fe0 con 0x7f64c806ce50 2024-04-25T19:46:02.704 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.702+0000 7f64ccb2a640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f64c8147da0 con 0x7f64c814af10 2024-04-25T19:46:02.704 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.702+0000 7f64c659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f64c806ce50 0x7f64c80a5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:02.704 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.702+0000 7f64c6d9d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f64c814af10 0x7f64c8151840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:02.704 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.702+0000 7f64c659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f64c806ce50 0x7f64c80a5f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51198/0 (socket says 172.21.15.29:51198) 2024-04-25T19:46:02.704 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.702+0000 7f64c659c640 1 -- 172.21.15.29:0/2892073725 learned_addr learned my addr 172.21.15.29:0/2892073725 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:46:02.705 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.703+0000 7f64c659c640 1 -- 172.21.15.29:0/2892073725 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f64c814af10 msgr2=0x7f64c8151840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:46:02.705 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.703+0000 7f64c659c640 1 --2- 172.21.15.29:0/2892073725 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f64c814af10 0x7f64c8151840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.705 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.703+0000 7f64c659c640 1 -- 172.21.15.29:0/2892073725 >> v1:172.21.15.102:6789/0 conn(0x7f64c80a6440 legacy=0x7f64c814a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:02.705 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.703+0000 7f64c659c640 1 -- 172.21.15.29:0/2892073725 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64c80a7210 con 0x7f64c806ce50 2024-04-25T19:46:02.706 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.704+0000 7f64c659c640 1 --2- 172.21.15.29:0/2892073725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64c806ce50 0x7f64c80a5f00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f64b8009850 tx=0x7f64b802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7702a26ebfe38368 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:02.706 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.704+0000 7f64c559a640 1 -- 172.21.15.29:0/2892073725 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b8004230 con 0x7f64c806ce50 2024-04-25T19:46:02.706 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.704+0000 7f64c559a640 1 -- 172.21.15.29:0/2892073725 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64b80043d0 con 0x7f64c806ce50 2024-04-25T19:46:02.706 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.704+0000 7f64c559a640 1 -- 172.21.15.29:0/2892073725 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b8003810 con 0x7f64c806ce50 2024-04-25T19:46:02.706 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.704+0000 7f64ccb2a640 1 -- 172.21.15.29:0/2892073725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64c806ce50 msgr2=0x7f64c80a5f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:02.706 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.704+0000 7f64ccb2a640 1 --2- 172.21.15.29:0/2892073725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64c806ce50 0x7f64c80a5f00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f64b8009850 tx=0x7f64b802b9f0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.706 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.705+0000 7f64ccb2a640 1 -- 172.21.15.29:0/2892073725 shutdown_connections 2024-04-25T19:46:02.706 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.705+0000 7f64ccb2a640 1 --2- 172.21.15.29:0/2892073725 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f64c814af10 0x7f64c8151840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.707 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.705+0000 7f64ccb2a640 1 --2- 172.21.15.29:0/2892073725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64c806ce50 0x7f64c80a5f00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.707 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.705+0000 7f64ccb2a640 1 -- 172.21.15.29:0/2892073725 >> 172.21.15.29:0/2892073725 conn(0x7f64c806df60 msgr2=0x7f64c806e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:02.707 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.705+0000 7f64ccb2a640 1 -- 172.21.15.29:0/2892073725 shutdown_connections 2024-04-25T19:46:02.707 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.705+0000 7f64ccb2a640 1 -- 172.21.15.29:0/2892073725 wait complete. 2024-04-25T19:46:02.707 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.706+0000 7f64ccb2a640 1 Processor -- start 2024-04-25T19:46:02.707 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.706+0000 7f64ccb2a640 1 -- start start 2024-04-25T19:46:02.709 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.707+0000 7f64ccb2a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64c806ce50 0x7f64c8146690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:02.710 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.707+0000 7f64ccb2a640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f64c80a6440 0x7f64c813fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:02.710 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.707+0000 7f64ccb2a640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64c814af10 0x7f64c8145240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:02.710 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.707+0000 7f64ccb2a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f64c806d3e0 con 0x7f64c806ce50 2024-04-25T19:46:02.710 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.707+0000 7f64ccb2a640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f64c806d260 con 0x7f64c814af10 2024-04-25T19:46:02.710 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.707+0000 7f64ccb2a640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f64c8151d80 con 0x7f64c80a6440 2024-04-25T19:46:02.710 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.707+0000 7f64c5d9b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f64c80a6440 0x7f64c813fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:02.710 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.707+0000 7f64c659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64c806ce50 0x7f64c8146690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:02.710 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.707+0000 7f64c5d9b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f64c80a6440 0x7f64c813fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:49962/0 (socket says 172.21.15.29:49962) 2024-04-25T19:46:02.710 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.707+0000 7f64c5d9b640 1 -- 172.21.15.29:0/2698920919 learned_addr learned my addr 172.21.15.29:0/2698920919 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:46:02.710 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.707+0000 7f64c659c640 1 --2- 172.21.15.29:0/2698920919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64c806ce50 0x7f64c8146690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51210/0 (socket says 172.21.15.29:51210) 2024-04-25T19:46:02.710 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.707+0000 7f64c6d9d640 1 --2- 172.21.15.29:0/2698920919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64c814af10 0x7f64c8145240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:02.710 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.708+0000 7f64c5d9b640 1 -- 172.21.15.29:0/2698920919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64c814af10 msgr2=0x7f64c8145240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:02.710 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.708+0000 7f64c5d9b640 1 --2- 172.21.15.29:0/2698920919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64c814af10 0x7f64c8145240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.711 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.708+0000 7f64c5d9b640 1 -- 172.21.15.29:0/2698920919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64c806ce50 msgr2=0x7f64c8146690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:02.711 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.708+0000 7f64c5d9b640 1 --2- 172.21.15.29:0/2698920919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64c806ce50 0x7f64c8146690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.711 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.708+0000 7f64c5d9b640 1 -- 172.21.15.29:0/2698920919 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64c813efa0 con 0x7f64c80a6440 2024-04-25T19:46:02.711 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.708+0000 7f64c659c640 1 --2- 172.21.15.29:0/2698920919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64c806ce50 0x7f64c8146690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:46:02.711 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.708+0000 7f64c5d9b640 1 --2- 172.21.15.29:0/2698920919 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f64c80a6440 0x7f64c813fd20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f64bc00c420 tx=0x7f64bc00c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:02.711 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.708+0000 7f64b6ffd640 1 -- 172.21.15.29:0/2698920919 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64bc0052d0 con 0x7f64c80a6440 2024-04-25T19:46:02.711 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.708+0000 7f64b6ffd640 1 -- 172.21.15.29:0/2698920919 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64bc005490 con 0x7f64c80a6440 2024-04-25T19:46:02.711 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.708+0000 7f64b6ffd640 1 -- 172.21.15.29:0/2698920919 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64bc007260 con 0x7f64c80a6440 2024-04-25T19:46:02.711 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.708+0000 7f64ccb2a640 1 -- 172.21.15.29:0/2698920919 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64c813eb80 con 0x7f64c80a6440 2024-04-25T19:46:02.711 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.708+0000 7f64ccb2a640 1 -- 172.21.15.29:0/2698920919 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f64c813e780 con 0x7f64c80a6440 2024-04-25T19:46:02.712 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.710+0000 7f64ccb2a640 1 -- 172.21.15.29:0/2698920919 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64c80692d0 con 0x7f64c80a6440 2024-04-25T19:46:02.712 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.711+0000 7f64b6ffd640 1 -- 172.21.15.29:0/2698920919 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 12) ==== 144961+0+0 (secure 0 0 0) 0x7f64bc007400 con 0x7f64c80a6440 2024-04-25T19:46:02.712 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.711+0000 7f64b6ffd640 20 mgrc handle_mgr_map mgrmap(e 12) 2024-04-25T19:46:02.713 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.711+0000 7f64b6ffd640 4 mgrc handle_mgr_map Got map version 12 2024-04-25T19:46:02.713 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.711+0000 7f64b6ffd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] 2024-04-25T19:46:02.713 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.711+0000 7f64b6ffd640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] 2024-04-25T19:46:02.713 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.711+0000 7f64b6ffd640 1 --2- 172.21.15.29:0/2698920919 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f64a80ac680 0x7f64a80aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:02.713 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.711+0000 7f64c659c640 1 -- 172.21.15.29:0/2698920919 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f64a80ac680 msgr2=0x7f64a80aeb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6824/1500823528 2024-04-25T19:46:02.713 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.712+0000 7f64b6ffd640 1 -- 172.21.15.29:0/2698920919 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 4854+0+0 (secure 0 0 0) 0x7f64bc0d8a70 con 0x7f64c80a6440 2024-04-25T19:46:02.713 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.712+0000 7f64c659c640 1 --2- 172.21.15.29:0/2698920919 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f64a80ac680 0x7f64a80aeb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T19:46:02.717 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.716+0000 7f64b6ffd640 1 -- 172.21.15.29:0/2698920919 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f64bc0a2d70 con 0x7f64c80a6440 2024-04-25T19:46:02.775 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:02.775+0000 7f4e1e83c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T19:46:02.778 INFO:tasks.ceph.mgr.z.smithi102.stderr:2024-04-25T19:46:02.778+0000 7f846c637040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T19:46:02.790 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.789+0000 7f64b6ffd640 1 -- 172.21.15.29:0/2698920919 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mgrmap(e 13) ==== 144570+0+0 (secure 0 0 0) 0x7f64bc0d72e0 con 0x7f64c80a6440 2024-04-25T19:46:02.790 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.789+0000 7f64b6ffd640 20 mgrc handle_mgr_map mgrmap(e 13) 2024-04-25T19:46:02.790 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.789+0000 7f64b6ffd640 4 mgrc handle_mgr_map Got map version 13 2024-04-25T19:46:02.791 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.789+0000 7f64b6ffd640 4 mgrc handle_mgr_map Active mgr is now 2024-04-25T19:46:02.791 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.789+0000 7f64b6ffd640 4 mgrc reconnect Terminating session with v2:172.21.15.102:6824/1500823528 2024-04-25T19:46:02.791 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.789+0000 7f64b6ffd640 1 -- 172.21.15.29:0/2698920919 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f64a80ac680 msgr2=0x7f64a80aeb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T19:46:02.791 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.789+0000 7f64b6ffd640 1 --2- 172.21.15.29:0/2698920919 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f64a80ac680 0x7f64a80aeb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:02.791 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.789+0000 7f64b6ffd640 4 mgrc reconnect No active mgr available yet 2024-04-25T19:46:02.841 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-04-25T19:46:02.839+0000 7f33fe212040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T19:46:02.843 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:02.842+0000 7f4da0aa4640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:46:02.843 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:02.842+0000 7f4da0aa4640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:46:02.843 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:02.842+0000 7f4da0aa4640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:46:02.843 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:02.842+0000 7f4da0aa4640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:46:02.843 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:02.842+0000 7f4da0aa4640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:46:02.843 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:02.842+0000 7f4da3aaa640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:46:02.843 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:02.842+0000 7f4da3aaa640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:46:02.843 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:02.842+0000 7f4da3aaa640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:46:02.843 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:02.842+0000 7f4da3aaa640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:46:02.844 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-04-25T19:46:02.842+0000 7f4da3aaa640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:46:02.925 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.924+0000 7f64ccb2a640 1 -- 172.21.15.29:0/2698920919 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/localpool/failure_domain"} v 0) -- 0x7f64c80b8d90 con 0x7f64c80a6440 2024-04-25T19:46:02.929 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:02.928+0000 7f64b6ffd640 1 -- 172.21.15.29:0/2698920919 <== mon.2 v2:172.21.15.29:3301/0 8 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/localpool/failure_domain"}]=0 v9) ==== 111+0+0 (secure 0 0 0) 0x7f64bc078bf0 con 0x7f64c80a6440 2024-04-25T19:46:03.038 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:03.037+0000 7f64ccb2a640 1 -- 172.21.15.29:0/2698920919 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/localpool/failure_domain"} v 0) -- 0x7f64c80b6390 con 0x7f64c80a6440 2024-04-25T19:46:03.039 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:03.038+0000 7f64b6ffd640 1 -- 172.21.15.29:0/2698920919 <== mon.2 v2:172.21.15.29:3301/0 9 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/localpool/failure_domain"}]=0 v9) ==== 111+0+0 (secure 0 0 0) 0x7f64bc08d400 con 0x7f64c80a6440 2024-04-25T19:46:03.041 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:03.040+0000 7f64ccb2a640 10 mgrc shutdown 2024-04-25T19:46:03.041 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:03.040+0000 7f64ccb2a640 1 -- 172.21.15.29:0/2698920919 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f64c80a6440 msgr2=0x7f64c813fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:03.041 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:03.040+0000 7f64ccb2a640 1 --2- 172.21.15.29:0/2698920919 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f64c80a6440 0x7f64c813fd20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f64bc00c420 tx=0x7f64bc00c8e0 comp rx=0 tx=0).stop 2024-04-25T19:46:03.041 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:03.040+0000 7f64ccb2a640 1 -- 172.21.15.29:0/2698920919 shutdown_connections 2024-04-25T19:46:03.042 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:03.040+0000 7f64ccb2a640 1 --2- 172.21.15.29:0/2698920919 >> [v2:172.21.15.102:6824/1500823528,v1:172.21.15.102:6825/1500823528] conn(0x7f64a80ac680 0x7f64a80aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:03.042 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:03.040+0000 7f64ccb2a640 1 --2- 172.21.15.29:0/2698920919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64c814af10 0x7f64c8145240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:03.042 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:03.040+0000 7f64ccb2a640 1 --2- 172.21.15.29:0/2698920919 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f64c80a6440 0x7f64c813fd20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:03.042 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:03.040+0000 7f64ccb2a640 1 --2- 172.21.15.29:0/2698920919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64c806ce50 0x7f64c8146690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:03.042 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:03.040+0000 7f64ccb2a640 1 -- 172.21.15.29:0/2698920919 >> 172.21.15.29:0/2698920919 conn(0x7f64c806df60 msgr2=0x7f64c8149c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:03.042 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:03.040+0000 7f64ccb2a640 1 -- 172.21.15.29:0/2698920919 shutdown_connections 2024-04-25T19:46:03.042 INFO:tasks.workunit.client.0.smithi029.stderr:2024-04-25T19:46:03.040+0000 7f64ccb2a640 1 -- 172.21.15.29:0/2698920919 wait complete. 2024-04-25T19:46:03.050 INFO:tasks.workunit.client.0.smithi029.stdout:OK 2024-04-25T19:46:03.051 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo OK 2024-04-25T19:46:03.051 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T19:46:03.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T19:46:03.116 INFO:tasks.workunit:Stopping ['mgr/test_localpool.sh'] on client.0... 2024-04-25T19:46:03.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-25T19:46:03.414 DEBUG:teuthology.parallel:result is None 2024-04-25T19:46:03.415 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T19:46:03.439 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T19:46:03.439 DEBUG:teuthology.orchestra.run.smithi029:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-04-25T19:46:03.500 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T19:46:03.500 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-25T19:46:03.510 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-25T19:46:03.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.649+0000 7f3498044640 1 Processor -- start 2024-04-25T19:46:03.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.649+0000 7f3498044640 1 -- start start 2024-04-25T19:46:03.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.650+0000 7f3498044640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3490143e30 0x7f3490144240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:03.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.650+0000 7f3498044640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f34900acaa0 0x7f34900aceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:03.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.650+0000 7f3498044640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34900a5a90 con 0x7f3490146c10 2024-04-25T19:46:03.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.650+0000 7f3498044640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f3490147790 con 0x7f3490143e30 2024-04-25T19:46:03.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.650+0000 7f3498044640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f3490147a50 con 0x7f34900acaa0 2024-04-25T19:46:03.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.650+0000 7f34955df640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f34900acaa0 0x7f34900aceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:03.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.650+0000 7f34965e1640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3490146c10 0x7f349014a200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43254/0 (socket says 172.21.15.29:43254) 2024-04-25T19:46:03.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.650+0000 7f34965e1640 1 -- 172.21.15.29:0/813715156 learned_addr learned my addr 172.21.15.29:0/813715156 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:46:03.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.650+0000 7f34955df640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f34900acaa0 0x7f34900aceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:49978/0 (socket says 172.21.15.29:49978) 2024-04-25T19:46:03.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.650+0000 7f3495de0640 1 --2- 172.21.15.29:0/813715156 >> v2:172.21.15.102:3300/0 conn(0x7f3490143e30 0x7f3490144240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:03.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.651+0000 7f3494dde640 1 -- 172.21.15.29:0/813715156 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1194820047 0 0) 0x7f34900a5a90 con 0x7f3490146c10 2024-04-25T19:46:03.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.651+0000 7f34955df640 1 -- 172.21.15.29:0/813715156 >> v2:172.21.15.102:3300/0 conn(0x7f3490143e30 msgr2=0x7f3490144240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:46:03.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.651+0000 7f34955df640 1 --2- 172.21.15.29:0/813715156 >> v2:172.21.15.102:3300/0 conn(0x7f3490143e30 0x7f3490144240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:03.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.651+0000 7f34955df640 1 -- 172.21.15.29:0/813715156 >> v1:172.21.15.29:6789/0 conn(0x7f3490146c10 legacy=0x7f349014a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:03.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.651+0000 7f34955df640 1 -- 172.21.15.29:0/813715156 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3490147020 con 0x7f34900acaa0 2024-04-25T19:46:03.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.651+0000 7f34955df640 1 --2- 172.21.15.29:0/813715156 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f34900acaa0 0x7f34900aceb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f34800097d0 tx=0x7f348002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ae4b9bed12bd20b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:03.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.651+0000 7f3494dde640 1 -- 172.21.15.29:0/813715156 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3480035020 con 0x7f34900acaa0 2024-04-25T19:46:03.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.652+0000 7f3494dde640 1 -- 172.21.15.29:0/813715156 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3480031800 con 0x7f34900acaa0 2024-04-25T19:46:03.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.652+0000 7f3494dde640 1 -- 172.21.15.29:0/813715156 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3480031bd0 con 0x7f34900acaa0 2024-04-25T19:46:03.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.652+0000 7f3498044640 1 -- 172.21.15.29:0/813715156 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f34900acaa0 msgr2=0x7f34900aceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:03.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.652+0000 7f3498044640 1 --2- 172.21.15.29:0/813715156 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f34900acaa0 0x7f34900aceb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f34800097d0 tx=0x7f348002b8a0 comp rx=0 tx=0).stop 2024-04-25T19:46:03.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.652+0000 7f3498044640 1 -- 172.21.15.29:0/813715156 shutdown_connections 2024-04-25T19:46:03.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.652+0000 7f3498044640 1 --2- 172.21.15.29:0/813715156 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f34900acaa0 0x7f34900aceb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:03.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.652+0000 7f3498044640 1 --2- 172.21.15.29:0/813715156 >> v2:172.21.15.102:3300/0 conn(0x7f3490143e30 0x7f3490144240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:03.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.652+0000 7f3498044640 1 -- 172.21.15.29:0/813715156 >> 172.21.15.29:0/813715156 conn(0x7f34900aafc0 msgr2=0x7f34900ab3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:03.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.652+0000 7f3498044640 1 -- 172.21.15.29:0/813715156 shutdown_connections 2024-04-25T19:46:03.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.652+0000 7f3498044640 1 -- 172.21.15.29:0/813715156 wait complete. 2024-04-25T19:46:03.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.654+0000 7f3498044640 1 Processor -- start 2024-04-25T19:46:03.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.654+0000 7f3498044640 1 -- start start 2024-04-25T19:46:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.655+0000 7f3498044640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34900acaa0 0x7f34901d2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.655+0000 7f3498044640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3490143e30 0x7f3490142670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.655+0000 7f3498044640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3490146c10 0x7f34901e1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.655+0000 7f3498044640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f349014aa90 con 0x7f34900acaa0 2024-04-25T19:46:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.655+0000 7f3498044640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f349014a910 con 0x7f3490143e30 2024-04-25T19:46:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.655+0000 7f3498044640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f349014ac10 con 0x7f3490146c10 2024-04-25T19:46:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.655+0000 7f3495de0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34900acaa0 0x7f34901d2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.655+0000 7f34965e1640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3490146c10 0x7f34901e1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.655+0000 7f3495de0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34900acaa0 0x7f34901d2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51224/0 (socket says 172.21.15.29:51224) 2024-04-25T19:46:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.655+0000 7f3495de0640 1 -- 172.21.15.29:0/4013123819 learned_addr learned my addr 172.21.15.29:0/4013123819 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:46:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.655+0000 7f34955df640 1 --2- 172.21.15.29:0/4013123819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3490143e30 0x7f3490142670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.655+0000 7f34965e1640 1 -- 172.21.15.29:0/4013123819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3490143e30 msgr2=0x7f3490142670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.655+0000 7f34965e1640 1 --2- 172.21.15.29:0/4013123819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3490143e30 0x7f3490142670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.655+0000 7f34965e1640 1 -- 172.21.15.29:0/4013123819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34900acaa0 msgr2=0x7f34901d2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.655+0000 7f34965e1640 1 --2- 172.21.15.29:0/4013123819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34900acaa0 0x7f34901d2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.655+0000 7f34965e1640 1 -- 172.21.15.29:0/4013123819 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3490143810 con 0x7f3490146c10 2024-04-25T19:46:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.655+0000 7f3495de0640 1 --2- 172.21.15.29:0/4013123819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34900acaa0 0x7f34901d2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:46:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.655+0000 7f34955df640 1 --2- 172.21.15.29:0/4013123819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3490143e30 0x7f3490142670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:46:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.655+0000 7f34965e1640 1 --2- 172.21.15.29:0/4013123819 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3490146c10 0x7f34901e1140 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f348800c750 tx=0x7f348800cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.656+0000 7f34867fc640 1 -- 172.21.15.29:0/4013123819 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3488018070 con 0x7f3490146c10 2024-04-25T19:46:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.656+0000 7f34867fc640 1 -- 172.21.15.29:0/4013123819 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3488004d60 con 0x7f3490146c10 2024-04-25T19:46:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.656+0000 7f3498044640 1 -- 172.21.15.29:0/4013123819 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3490142bb0 con 0x7f3490146c10 2024-04-25T19:46:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.656+0000 7f34867fc640 1 -- 172.21.15.29:0/4013123819 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348800a880 con 0x7f3490146c10 2024-04-25T19:46:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.656+0000 7f3498044640 1 -- 172.21.15.29:0/4013123819 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3490145bf0 con 0x7f3490146c10 2024-04-25T19:46:03.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.657+0000 7f3498044640 1 -- 172.21.15.29:0/4013123819 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3490142f00 con 0x7f3490146c10 2024-04-25T19:46:03.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.658+0000 7f34867fc640 1 -- 172.21.15.29:0/4013123819 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 13) ==== 144570+0+0 (secure 0 0 0) 0x7f348801b070 con 0x7f3490146c10 2024-04-25T19:46:03.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.658+0000 7f34867fc640 20 mgrc handle_mgr_map mgrmap(e 13) 2024-04-25T19:46:03.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.659+0000 7f34867fc640 4 mgrc handle_mgr_map Got map version 13 2024-04-25T19:46:03.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.659+0000 7f34867fc640 4 mgrc handle_mgr_map Active mgr is now 2024-04-25T19:46:03.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.659+0000 7f34867fc640 4 mgrc reconnect No active mgr available yet 2024-04-25T19:46:03.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.659+0000 7f34867fc640 1 -- 172.21.15.29:0/4013123819 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(31..31 src has 1..31) ==== 5198+0+0 (secure 0 0 0) 0x7f34880d85a0 con 0x7f3490146c10 2024-04-25T19:46:03.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.663+0000 7f34867fc640 1 -- 172.21.15.29:0/4013123819 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3490142f00 con 0x7f3490146c10 2024-04-25T19:46:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.790+0000 7f34867fc640 1 -- 172.21.15.29:0/4013123819 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mgrmap(e 14) ==== 144697+0+0 (secure 0 0 0) 0x7f348809e3f0 con 0x7f3490146c10 2024-04-25T19:46:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.790+0000 7f34867fc640 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-25T19:46:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.791+0000 7f34867fc640 4 mgrc handle_mgr_map Got map version 14 2024-04-25T19:46:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.791+0000 7f34867fc640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] 2024-04-25T19:46:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.791+0000 7f34867fc640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] 2024-04-25T19:46:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.791+0000 7f34867fc640 1 --2- 172.21.15.29:0/4013123819 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f34740b1920 0x7f34740b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.792+0000 7f3495de0640 1 --2- 172.21.15.29:0/4013123819 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f34740b1920 0x7f34740b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.793+0000 7f3495de0640 1 --2- 172.21.15.29:0/4013123819 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f34740b1920 0x7f34740b3d40 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f348c009f50 tx=0x7f348c009190 comp rx=0 tx=0).ready entity=mgr.4595 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:03.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.857+0000 7f3498044640 1 -- 172.21.15.29:0/4013123819 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f34900a6ab0 con 0x7f3490146c10 2024-04-25T19:46:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.861+0000 7f34867fc640 1 -- 172.21.15.29:0/4013123819 <== mon.2 v2:172.21.15.29:3301/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f34880a2620 con 0x7f3490146c10 2024-04-25T19:46:03.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.863+0000 7f34867fc640 1 -- 172.21.15.29:0/4013123819 <== mon.2 v2:172.21.15.29:3301/0 9 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) ==== 123+0+0 (secure 0 0 0) 0x7f34880a3210 con 0x7f3490146c10 2024-04-25T19:46:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.865+0000 7f3498044640 10 mgrc shutdown 2024-04-25T19:46:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.865+0000 7f3498044640 1 -- 172.21.15.29:0/4013123819 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f34740b1920 msgr2=0x7f34740b3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.865+0000 7f3498044640 1 --2- 172.21.15.29:0/4013123819 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f34740b1920 0x7f34740b3d40 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f348c009f50 tx=0x7f348c009190 comp rx=0 tx=0).stop 2024-04-25T19:46:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.865+0000 7f3498044640 1 -- 172.21.15.29:0/4013123819 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3490146c10 msgr2=0x7f34901e1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.865+0000 7f3498044640 1 --2- 172.21.15.29:0/4013123819 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3490146c10 0x7f34901e1140 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f348800c750 tx=0x7f348800cc10 comp rx=0 tx=0).stop 2024-04-25T19:46:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.865+0000 7f3498044640 1 -- 172.21.15.29:0/4013123819 shutdown_connections 2024-04-25T19:46:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.865+0000 7f3498044640 1 --2- 172.21.15.29:0/4013123819 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f34740b1920 0x7f34740b3d40 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.865+0000 7f3498044640 1 --2- 172.21.15.29:0/4013123819 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3490146c10 0x7f34901e1140 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.865+0000 7f3498044640 1 --2- 172.21.15.29:0/4013123819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3490143e30 0x7f3490142670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.865+0000 7f3498044640 1 --2- 172.21.15.29:0/4013123819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34900acaa0 0x7f34901d2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.865+0000 7f3498044640 1 -- 172.21.15.29:0/4013123819 >> 172.21.15.29:0/4013123819 conn(0x7f34900aafc0 msgr2=0x7f34901d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.865+0000 7f3498044640 1 -- 172.21.15.29:0/4013123819 shutdown_connections 2024-04-25T19:46:03.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.865+0000 7f3498044640 1 -- 172.21.15.29:0/4013123819 wait complete. 2024-04-25T19:46:03.877 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T19:46:03.878 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:46:03.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.989+0000 7fa210bfa640 1 Processor -- start 2024-04-25T19:46:03.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.989+0000 7fa210bfa640 1 -- start start 2024-04-25T19:46:03.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.990+0000 7fa210bfa640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa20c137650 0x7fa20c0b7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:03.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.990+0000 7fa210bfa640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa20c0ba510 con 0x7fa20c0ba100 2024-04-25T19:46:03.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.990+0000 7fa210bfa640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa20c137bd0 con 0x7fa20c0b70f0 2024-04-25T19:46:03.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.990+0000 7fa210bfa640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fa20c137e90 con 0x7fa20c137650 2024-04-25T19:46:03.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.990+0000 7fa20a59c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa20c0ba100 0x7fa20c0addb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43262/0 (socket says 172.21.15.29:43262) 2024-04-25T19:46:03.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.990+0000 7fa20ad9d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa20c137650 0x7fa20c0b7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:03.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.990+0000 7fa20a59c640 1 -- 172.21.15.29:0/3817746394 learned_addr learned my addr 172.21.15.29:0/3817746394 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:46:03.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.990+0000 7fa20ad9d640 1 -- 172.21.15.29:0/3817746394 >> v1:172.21.15.102:6789/0 conn(0x7fa20c0b70f0 legacy=0x7fa20c136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:03.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.990+0000 7fa20ad9d640 1 -- 172.21.15.29:0/3817746394 >> v1:172.21.15.29:6789/0 conn(0x7fa20c0ba100 legacy=0x7fa20c0addb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:03.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.990+0000 7fa209d9b640 1 -- 172.21.15.29:0/3817746394 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1806028423 0 0) 0x7fa1fc001110 con 0x7fa20c0ba100 2024-04-25T19:46:03.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.990+0000 7fa20ad9d640 1 -- 172.21.15.29:0/3817746394 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa20c152220 con 0x7fa20c137650 2024-04-25T19:46:03.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.991+0000 7fa20ad9d640 1 --2- 172.21.15.29:0/3817746394 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa20c137650 0x7fa20c0b7500 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa204009dc0 tx=0x7fa20402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8dbf7cb3762c4354 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:03.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.991+0000 7fa209d9b640 1 -- 172.21.15.29:0/3817746394 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa204039070 con 0x7fa20c137650 2024-04-25T19:46:03.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.991+0000 7fa209d9b640 1 -- 172.21.15.29:0/3817746394 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fa20402ccd0 con 0x7fa20c137650 2024-04-25T19:46:03.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.991+0000 7fa209d9b640 1 -- 172.21.15.29:0/3817746394 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa204034680 con 0x7fa20c137650 2024-04-25T19:46:03.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.992+0000 7fa210bfa640 1 -- 172.21.15.29:0/3817746394 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa20c137650 msgr2=0x7fa20c0b7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:03.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.992+0000 7fa210bfa640 1 --2- 172.21.15.29:0/3817746394 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa20c137650 0x7fa20c0b7500 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa204009dc0 tx=0x7fa20402bf20 comp rx=0 tx=0).stop 2024-04-25T19:46:03.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.992+0000 7fa210bfa640 1 -- 172.21.15.29:0/3817746394 shutdown_connections 2024-04-25T19:46:03.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.992+0000 7fa210bfa640 1 --2- 172.21.15.29:0/3817746394 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa20c137650 0x7fa20c0b7500 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:03.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.992+0000 7fa210bfa640 1 -- 172.21.15.29:0/3817746394 >> 172.21.15.29:0/3817746394 conn(0x7fa20c072b90 msgr2=0x7fa20c0a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:03.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.992+0000 7fa210bfa640 1 -- 172.21.15.29:0/3817746394 shutdown_connections 2024-04-25T19:46:03.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.992+0000 7fa210bfa640 1 -- 172.21.15.29:0/3817746394 wait complete. 2024-04-25T19:46:03.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.994+0000 7fa210bfa640 1 Processor -- start 2024-04-25T19:46:03.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.994+0000 7fa210bfa640 1 -- start start 2024-04-25T19:46:03.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa210bfa640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa20c0b70f0 0x7fa20c132030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:03.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa210bfa640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa20c0ba100 0x7fa20c12a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:03.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa210bfa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa20c137650 0x7fa20c14f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:03.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa210bfa640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa20c150640 con 0x7fa20c137650 2024-04-25T19:46:03.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa210bfa640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa20c1504c0 con 0x7fa20c0ba100 2024-04-25T19:46:03.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa210bfa640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fa20c1507c0 con 0x7fa20c0b70f0 2024-04-25T19:46:03.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa20a59c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa20c0b70f0 0x7fa20c132030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:03.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa20a59c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa20c0b70f0 0x7fa20c132030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:49992/0 (socket says 172.21.15.29:49992) 2024-04-25T19:46:03.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa20ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa20c137650 0x7fa20c14f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:03.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa20a59c640 1 -- 172.21.15.29:0/1638994823 learned_addr learned my addr 172.21.15.29:0/1638994823 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:46:03.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa203fff640 1 --2- 172.21.15.29:0/1638994823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa20c0ba100 0x7fa20c12a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:03.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa20a59c640 1 -- 172.21.15.29:0/1638994823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa20c0ba100 msgr2=0x7fa20c12a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:03.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa20a59c640 1 --2- 172.21.15.29:0/1638994823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa20c0ba100 0x7fa20c12a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:03.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa20a59c640 1 -- 172.21.15.29:0/1638994823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa20c137650 msgr2=0x7fa20c14f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:03.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa20a59c640 1 --2- 172.21.15.29:0/1638994823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa20c137650 0x7fa20c14f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:03.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa20a59c640 1 -- 172.21.15.29:0/1638994823 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa20c129ed0 con 0x7fa20c0b70f0 2024-04-25T19:46:03.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa20ad9d640 1 --2- 172.21.15.29:0/1638994823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa20c137650 0x7fa20c14f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:46:03.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa20a59c640 1 --2- 172.21.15.29:0/1638994823 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa20c0b70f0 0x7fa20c132030 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc00c3c0 tx=0x7fa1fc00c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:03.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa202ffd640 1 -- 172.21.15.29:0/1638994823 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc0074e0 con 0x7fa20c0b70f0 2024-04-25T19:46:03.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa202ffd640 1 -- 172.21.15.29:0/1638994823 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fa1fc0047e0 con 0x7fa20c0b70f0 2024-04-25T19:46:03.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa210bfa640 1 -- 172.21.15.29:0/1638994823 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa20c12b740 con 0x7fa20c0b70f0 2024-04-25T19:46:03.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa202ffd640 1 -- 172.21.15.29:0/1638994823 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc004bf0 con 0x7fa20c0b70f0 2024-04-25T19:46:03.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.995+0000 7fa210bfa640 1 -- 172.21.15.29:0/1638994823 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa20c132570 con 0x7fa20c0b70f0 2024-04-25T19:46:03.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.996+0000 7fa210bfa640 1 -- 172.21.15.29:0/1638994823 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa20c12b120 con 0x7fa20c0b70f0 2024-04-25T19:46:03.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.998+0000 7fa202ffd640 1 -- 172.21.15.29:0/1638994823 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 14) ==== 144697+0+0 (secure 0 0 0) 0x7fa1fc011070 con 0x7fa20c0b70f0 2024-04-25T19:46:03.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.998+0000 7fa202ffd640 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-25T19:46:03.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.998+0000 7fa202ffd640 4 mgrc handle_mgr_map Got map version 14 2024-04-25T19:46:03.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.998+0000 7fa202ffd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] 2024-04-25T19:46:03.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.998+0000 7fa202ffd640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] 2024-04-25T19:46:04.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.998+0000 7fa202ffd640 1 --2- 172.21.15.29:0/1638994823 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7fa1ec0ac380 0x7fa1ec0ae890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:04.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.999+0000 7fa202ffd640 1 -- 172.21.15.29:0/1638994823 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(31..31 src has 1..31) ==== 5198+0+0 (secure 0 0 0) 0x7fa1fc0d8e70 con 0x7fa20c0b70f0 2024-04-25T19:46:04.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.999+0000 7fa203fff640 1 --2- 172.21.15.29:0/1638994823 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7fa1ec0ac380 0x7fa1ec0ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:04.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:03.999+0000 7fa203fff640 1 --2- 172.21.15.29:0/1638994823 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7fa1ec0ac380 0x7fa1ec0ae890 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa20c12fa70 tx=0x7fa1f4009040 comp rx=0 tx=0).ready entity=mgr.4595 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:04.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.003+0000 7fa202ffd640 1 -- 172.21.15.29:0/1638994823 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa20c0ba510 con 0x7fa20c0b70f0 2024-04-25T19:46:04.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.226+0000 7fa210bfa640 1 -- 172.21.15.29:0/1638994823 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa20c0507b0 con 0x7fa20c0b70f0 2024-04-25T19:46:04.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.227+0000 7fa202ffd640 1 -- 172.21.15.29:0/1638994823 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+4351 (secure 0 0 0) 0x7fa1fc0a3ac0 con 0x7fa20c0b70f0 2024-04-25T19:46:04.228 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:46:04.228 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6809/3582557422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3582557422},{"type":"v1","addr":"172.21.15.29:6809","nonce":3582557422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4234,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/1228868047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1228868047},{"type":"v1","addr":"172.21.15.102:6827","nonce":1228868047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:44:09.410562+0000","modified":"2024-04-25T19:44:12.425418+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6811/844243878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":844243878},{"type":"v1","addr":"172.21.15.29:6811","nonce":844243878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":1}]} 2024-04-25T19:46:04.229 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2024-04-25T19:46:04.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.229+0000 7fa210bfa640 10 mgrc shutdown 2024-04-25T19:46:04.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.229+0000 7fa210bfa640 1 -- 172.21.15.29:0/1638994823 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7fa1ec0ac380 msgr2=0x7fa1ec0ae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:04.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.229+0000 7fa210bfa640 1 --2- 172.21.15.29:0/1638994823 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7fa1ec0ac380 0x7fa1ec0ae890 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa20c12fa70 tx=0x7fa1f4009040 comp rx=0 tx=0).stop 2024-04-25T19:46:04.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.229+0000 7fa210bfa640 1 -- 172.21.15.29:0/1638994823 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa20c0b70f0 msgr2=0x7fa20c132030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:04.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.229+0000 7fa210bfa640 1 --2- 172.21.15.29:0/1638994823 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa20c0b70f0 0x7fa20c132030 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc00c3c0 tx=0x7fa1fc00c880 comp rx=0 tx=0).stop 2024-04-25T19:46:04.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.229+0000 7fa210bfa640 1 -- 172.21.15.29:0/1638994823 shutdown_connections 2024-04-25T19:46:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.229+0000 7fa210bfa640 1 --2- 172.21.15.29:0/1638994823 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7fa1ec0ac380 0x7fa1ec0ae890 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.229+0000 7fa210bfa640 1 --2- 172.21.15.29:0/1638994823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa20c137650 0x7fa20c14f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.229+0000 7fa210bfa640 1 --2- 172.21.15.29:0/1638994823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa20c0ba100 0x7fa20c12a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.229+0000 7fa210bfa640 1 --2- 172.21.15.29:0/1638994823 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa20c0b70f0 0x7fa20c132030 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.229+0000 7fa210bfa640 1 -- 172.21.15.29:0/1638994823 >> 172.21.15.29:0/1638994823 conn(0x7fa20c072b90 msgr2=0x7fa20c0aace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.229+0000 7fa210bfa640 1 -- 172.21.15.29:0/1638994823 shutdown_connections 2024-04-25T19:46:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.229+0000 7fa210bfa640 1 -- 172.21.15.29:0/1638994823 wait complete. 2024-04-25T19:46:04.242 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:46:04.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.351+0000 7fcfceeaa640 1 Processor -- start 2024-04-25T19:46:04.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.351+0000 7fcfceeaa640 1 -- start start 2024-04-25T19:46:04.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.352+0000 7fcfceeaa640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcfc8144240 0x7fcfc8146c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:04.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.352+0000 7fcfceeaa640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcfc8147150 0x7fcfc8151590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:04.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.352+0000 7fcfceeaa640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfc8144650 con 0x7fcfc8143e30 2024-04-25T19:46:04.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.352+0000 7fcfceeaa640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fcfc8147560 con 0x7fcfc8144240 2024-04-25T19:46:04.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.352+0000 7fcfceeaa640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fcfc80a5a90 con 0x7fcfc8147150 2024-04-25T19:46:04.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.352+0000 7fcfcd447640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcfc8147150 0x7fcfc8151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:04.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.352+0000 7fcfcd447640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcfc8147150 0x7fcfc8151590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:50006/0 (socket says 172.21.15.29:50006) 2024-04-25T19:46:04.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.352+0000 7fcfcd447640 1 -- 172.21.15.29:0/2944346153 learned_addr learned my addr 172.21.15.29:0/2944346153 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:46:04.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.352+0000 7fcfc7fff640 1 --2- 172.21.15.29:0/2944346153 >> v2:172.21.15.102:3300/0 conn(0x7fcfc8144240 0x7fcfc8146c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:04.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.352+0000 7fcfcd447640 1 -- 172.21.15.29:0/2944346153 >> v2:172.21.15.102:3300/0 conn(0x7fcfc8144240 msgr2=0x7fcfc8146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:46:04.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.352+0000 7fcfcd447640 1 --2- 172.21.15.29:0/2944346153 >> v2:172.21.15.102:3300/0 conn(0x7fcfc8144240 0x7fcfc8146c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.352+0000 7fcfcd447640 1 -- 172.21.15.29:0/2944346153 >> v1:172.21.15.102:6789/0 conn(0x7fcfc8143e30 legacy=0x7fcfc80acaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:46:04.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.352+0000 7fcfcd447640 1 -- 172.21.15.29:0/2944346153 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfc8147790 con 0x7fcfc8147150 2024-04-25T19:46:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.353+0000 7fcfcd447640 1 --2- 172.21.15.29:0/2944346153 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcfc8147150 0x7fcfc8151590 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009e50 tx=0x7fcfc002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c5bcf68edad7090 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.353+0000 7fcfc77fe640 1 -- 172.21.15.29:0/2944346153 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc002ca70 con 0x7fcfc8147150 2024-04-25T19:46:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.353+0000 7fcfc77fe640 1 -- 172.21.15.29:0/2944346153 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fcfc002cc10 con 0x7fcfc8147150 2024-04-25T19:46:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.353+0000 7fcfc77fe640 1 -- 172.21.15.29:0/2944346153 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc00343f0 con 0x7fcfc8147150 2024-04-25T19:46:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.354+0000 7fcfceeaa640 1 -- 172.21.15.29:0/2944346153 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcfc8147150 msgr2=0x7fcfc8151590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.354+0000 7fcfceeaa640 1 --2- 172.21.15.29:0/2944346153 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcfc8147150 0x7fcfc8151590 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009e50 tx=0x7fcfc002bed0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.354+0000 7fcfceeaa640 1 -- 172.21.15.29:0/2944346153 shutdown_connections 2024-04-25T19:46:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.354+0000 7fcfceeaa640 1 --2- 172.21.15.29:0/2944346153 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcfc8147150 0x7fcfc8151590 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.354+0000 7fcfceeaa640 1 --2- 172.21.15.29:0/2944346153 >> v2:172.21.15.102:3300/0 conn(0x7fcfc8144240 0x7fcfc8146c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.354+0000 7fcfceeaa640 1 -- 172.21.15.29:0/2944346153 >> 172.21.15.29:0/2944346153 conn(0x7fcfc80aafc0 msgr2=0x7fcfc80ab3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.354+0000 7fcfceeaa640 1 -- 172.21.15.29:0/2944346153 shutdown_connections 2024-04-25T19:46:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.354+0000 7fcfceeaa640 1 -- 172.21.15.29:0/2944346153 wait complete. 2024-04-25T19:46:04.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.355+0000 7fcfceeaa640 1 Processor -- start 2024-04-25T19:46:04.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.356+0000 7fcfceeaa640 1 -- start start 2024-04-25T19:46:04.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.357+0000 7fcfceeaa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc8143e30 0x7fcfc8149de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:04.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.357+0000 7fcfceeaa640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfc8144240 0x7fcfc814e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:04.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.357+0000 7fcfceeaa640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcfc8147150 0x7fcfc81e07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:04.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.357+0000 7fcfceeaa640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc8151ad0 con 0x7fcfc8143e30 2024-04-25T19:46:04.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.357+0000 7fcfceeaa640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc80ad1f0 con 0x7fcfc8144240 2024-04-25T19:46:04.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.357+0000 7fcfceeaa640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fcfc8151c50 con 0x7fcfc8147150 2024-04-25T19:46:04.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.357+0000 7fcfcd447640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcfc8147150 0x7fcfc81e07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:04.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.357+0000 7fcfccc46640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc8143e30 0x7fcfc8149de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:04.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.357+0000 7fcfcd447640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcfc8147150 0x7fcfc81e07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:50014/0 (socket says 172.21.15.29:50014) 2024-04-25T19:46:04.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.357+0000 7fcfcd447640 1 -- 172.21.15.29:0/600464462 learned_addr learned my addr 172.21.15.29:0/600464462 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:46:04.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.357+0000 7fcfc7fff640 1 --2- 172.21.15.29:0/600464462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfc8144240 0x7fcfc814e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:04.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.357+0000 7fcfcd447640 1 -- 172.21.15.29:0/600464462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfc8144240 msgr2=0x7fcfc814e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:04.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.357+0000 7fcfcd447640 1 --2- 172.21.15.29:0/600464462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfc8144240 0x7fcfc814e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.357+0000 7fcfcd447640 1 -- 172.21.15.29:0/600464462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc8143e30 msgr2=0x7fcfc8149de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:04.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.357+0000 7fcfcd447640 1 --2- 172.21.15.29:0/600464462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc8143e30 0x7fcfc8149de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.357+0000 7fcfcd447640 1 -- 172.21.15.29:0/600464462 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfc81417f0 con 0x7fcfc8147150 2024-04-25T19:46:04.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.357+0000 7fcfccc46640 1 --2- 172.21.15.29:0/600464462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc8143e30 0x7fcfc8149de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:46:04.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.357+0000 7fcfcd447640 1 --2- 172.21.15.29:0/600464462 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcfc8147150 0x7fcfc81e07d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009e50 tx=0x7fcfc0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:04.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.357+0000 7fcfc57fa640 1 -- 172.21.15.29:0/600464462 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc0043020 con 0x7fcfc8147150 2024-04-25T19:46:04.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.358+0000 7fcfc57fa640 1 -- 172.21.15.29:0/600464462 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fcfc003e450 con 0x7fcfc8147150 2024-04-25T19:46:04.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.358+0000 7fcfceeaa640 1 -- 172.21.15.29:0/600464462 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfc814e420 con 0x7fcfc8147150 2024-04-25T19:46:04.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.358+0000 7fcfc57fa640 1 -- 172.21.15.29:0/600464462 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc003e880 con 0x7fcfc8147150 2024-04-25T19:46:04.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.358+0000 7fcfceeaa640 1 -- 172.21.15.29:0/600464462 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfc813e830 con 0x7fcfc8147150 2024-04-25T19:46:04.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.358+0000 7fcfceeaa640 1 -- 172.21.15.29:0/600464462 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfc8150ab0 con 0x7fcfc8147150 2024-04-25T19:46:04.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.361+0000 7fcfc57fa640 1 -- 172.21.15.29:0/600464462 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 14) ==== 144697+0+0 (secure 0 0 0) 0x7fcfc0046020 con 0x7fcfc8147150 2024-04-25T19:46:04.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.361+0000 7fcfc57fa640 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-25T19:46:04.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.361+0000 7fcfc57fa640 4 mgrc handle_mgr_map Got map version 14 2024-04-25T19:46:04.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.361+0000 7fcfc57fa640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] 2024-04-25T19:46:04.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.361+0000 7fcfc57fa640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] 2024-04-25T19:46:04.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.361+0000 7fcfc57fa640 1 --2- 172.21.15.29:0/600464462 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7fcfac0ac380 0x7fcfac0ae890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:04.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.361+0000 7fcfc57fa640 1 -- 172.21.15.29:0/600464462 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(31..31 src has 1..31) ==== 5198+0+0 (secure 0 0 0) 0x7fcfc00fc1b0 con 0x7fcfc8147150 2024-04-25T19:46:04.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.361+0000 7fcfccc46640 1 --2- 172.21.15.29:0/600464462 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7fcfac0ac380 0x7fcfac0ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:04.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.362+0000 7fcfccc46640 1 --2- 172.21.15.29:0/600464462 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7fcfac0ac380 0x7fcfac0ae890 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc006fd0 tx=0x7fcfbc009060 comp rx=0 tx=0).ready entity=mgr.4595 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:04.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.365+0000 7fcfc57fa640 1 -- 172.21.15.29:0/600464462 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcfc8150ab0 con 0x7fcfc8147150 2024-04-25T19:46:04.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.592+0000 7fcfceeaa640 1 -- 172.21.15.29:0/600464462 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcfc80507b0 con 0x7fcfc8147150 2024-04-25T19:46:04.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.593+0000 7fcfc57fa640 1 -- 172.21.15.29:0/600464462 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+4351 (secure 0 0 0) 0x7fcfc004d030 con 0x7fcfc8147150 2024-04-25T19:46:04.594 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:46:04.594 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6809/3582557422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3582557422},{"type":"v1","addr":"172.21.15.29:6809","nonce":3582557422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4234,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/1228868047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1228868047},{"type":"v1","addr":"172.21.15.102:6827","nonce":1228868047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:44:09.410562+0000","modified":"2024-04-25T19:44:12.425418+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6811/844243878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":844243878},{"type":"v1","addr":"172.21.15.29:6811","nonce":844243878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":1}]} 2024-04-25T19:46:04.594 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2024-04-25T19:46:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.595+0000 7fcfceeaa640 10 mgrc shutdown 2024-04-25T19:46:04.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.595+0000 7fcfceeaa640 1 -- 172.21.15.29:0/600464462 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7fcfac0ac380 msgr2=0x7fcfac0ae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:04.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.595+0000 7fcfceeaa640 1 --2- 172.21.15.29:0/600464462 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7fcfac0ac380 0x7fcfac0ae890 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc006fd0 tx=0x7fcfbc009060 comp rx=0 tx=0).stop 2024-04-25T19:46:04.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.595+0000 7fcfceeaa640 1 -- 172.21.15.29:0/600464462 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcfc8147150 msgr2=0x7fcfc81e07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:04.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.595+0000 7fcfceeaa640 1 --2- 172.21.15.29:0/600464462 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcfc8147150 0x7fcfc81e07d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009e50 tx=0x7fcfc0004a40 comp rx=0 tx=0).stop 2024-04-25T19:46:04.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.595+0000 7fcfceeaa640 1 -- 172.21.15.29:0/600464462 shutdown_connections 2024-04-25T19:46:04.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.595+0000 7fcfceeaa640 1 --2- 172.21.15.29:0/600464462 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7fcfac0ac380 0x7fcfac0ae890 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.595+0000 7fcfceeaa640 1 --2- 172.21.15.29:0/600464462 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcfc8147150 0x7fcfc81e07d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.595+0000 7fcfceeaa640 1 --2- 172.21.15.29:0/600464462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfc8144240 0x7fcfc814e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.595+0000 7fcfceeaa640 1 --2- 172.21.15.29:0/600464462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc8143e30 0x7fcfc8149de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.595+0000 7fcfceeaa640 1 -- 172.21.15.29:0/600464462 >> 172.21.15.29:0/600464462 conn(0x7fcfc80aafc0 msgr2=0x7fcfc80a6290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:04.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.595+0000 7fcfceeaa640 1 -- 172.21.15.29:0/600464462 shutdown_connections 2024-04-25T19:46:04.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.595+0000 7fcfceeaa640 1 -- 172.21.15.29:0/600464462 wait complete. 2024-04-25T19:46:04.608 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:46:04.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.717+0000 7fd80611b640 1 Processor -- start 2024-04-25T19:46:04.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.717+0000 7fd80611b640 1 -- start start 2024-04-25T19:46:04.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.718+0000 7fd80611b640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd800143e30 0x7fd800144240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:04.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.718+0000 7fd80611b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd8000aceb0 0x7fd800151590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:04.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.718+0000 7fd80611b640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8000a5a90 con 0x7fd8000acaa0 2024-04-25T19:46:04.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.718+0000 7fd80611b640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd800147790 con 0x7fd800143e30 2024-04-25T19:46:04.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.718+0000 7fd80611b640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fd800147a50 con 0x7fd8000aceb0 2024-04-25T19:46:04.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.718+0000 7fd7fffff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd8000aceb0 0x7fd800151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:04.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.718+0000 7fd7fffff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd8000aceb0 0x7fd800151590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:50018/0 (socket says 172.21.15.29:50018) 2024-04-25T19:46:04.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.718+0000 7fd7fffff640 1 -- 172.21.15.29:0/997728929 learned_addr learned my addr 172.21.15.29:0/997728929 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:46:04.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.719+0000 7fd7ff7fe640 1 --2- 172.21.15.29:0/997728929 >> v2:172.21.15.102:3300/0 conn(0x7fd800143e30 0x7fd800144240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:04.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.719+0000 7fd7fffff640 1 -- 172.21.15.29:0/997728929 >> v2:172.21.15.102:3300/0 conn(0x7fd800143e30 msgr2=0x7fd800144240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:46:04.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.719+0000 7fd7fffff640 1 --2- 172.21.15.29:0/997728929 >> v2:172.21.15.102:3300/0 conn(0x7fd800143e30 0x7fd800144240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.719+0000 7fd7fffff640 1 -- 172.21.15.29:0/997728929 >> v1:172.21.15.102:6789/0 conn(0x7fd8000acaa0 legacy=0x7fd800146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:46:04.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.719+0000 7fd7fffff640 1 -- 172.21.15.29:0/997728929 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8000ad2c0 con 0x7fd8000aceb0 2024-04-25T19:46:04.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.719+0000 7fd7fffff640 1 --2- 172.21.15.29:0/997728929 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd8000aceb0 0x7fd800151590 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd8000696e0 tx=0x7fd7f802be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1a44800aa58792c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:04.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.720+0000 7fd7fe7fc640 1 -- 172.21.15.29:0/997728929 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f802ca30 con 0x7fd8000aceb0 2024-04-25T19:46:04.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.720+0000 7fd7fe7fc640 1 -- 172.21.15.29:0/997728929 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fd7f802cbd0 con 0x7fd8000aceb0 2024-04-25T19:46:04.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.720+0000 7fd7fe7fc640 1 -- 172.21.15.29:0/997728929 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f80343f0 con 0x7fd8000aceb0 2024-04-25T19:46:04.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.720+0000 7fd80611b640 1 -- 172.21.15.29:0/997728929 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd8000aceb0 msgr2=0x7fd800151590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:04.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.720+0000 7fd80611b640 1 --2- 172.21.15.29:0/997728929 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd8000aceb0 0x7fd800151590 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd8000696e0 tx=0x7fd7f802be90 comp rx=0 tx=0).stop 2024-04-25T19:46:04.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.720+0000 7fd80611b640 1 -- 172.21.15.29:0/997728929 shutdown_connections 2024-04-25T19:46:04.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.720+0000 7fd80611b640 1 --2- 172.21.15.29:0/997728929 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd8000aceb0 0x7fd800151590 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.720+0000 7fd80611b640 1 --2- 172.21.15.29:0/997728929 >> v2:172.21.15.102:3300/0 conn(0x7fd800143e30 0x7fd800144240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.720+0000 7fd80611b640 1 -- 172.21.15.29:0/997728929 >> 172.21.15.29:0/997728929 conn(0x7fd8000aafc0 msgr2=0x7fd8000ab3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:04.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.720+0000 7fd80611b640 1 -- 172.21.15.29:0/997728929 shutdown_connections 2024-04-25T19:46:04.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.720+0000 7fd80611b640 1 -- 172.21.15.29:0/997728929 wait complete. 2024-04-25T19:46:04.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.722+0000 7fd80611b640 1 Processor -- start 2024-04-25T19:46:04.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.722+0000 7fd80611b640 1 -- start start 2024-04-25T19:46:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.723+0000 7fd80611b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8000acaa0 0x7fd800139fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.723+0000 7fd80611b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8000aceb0 0x7fd800144810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.723+0000 7fd80611b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd800143e30 0x7fd80009b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.723+0000 7fd80611b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd800151ad0 con 0x7fd8000acaa0 2024-04-25T19:46:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.723+0000 7fd80611b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd800147530 con 0x7fd8000aceb0 2024-04-25T19:46:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.723+0000 7fd80611b640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7fd800151c50 con 0x7fd800143e30 2024-04-25T19:46:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.723+0000 7fd7fffff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd800143e30 0x7fd80009b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.723+0000 7fd7ff7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8000acaa0 0x7fd800139fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.723+0000 7fd7fffff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd800143e30 0x7fd80009b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:50032/0 (socket says 172.21.15.29:50032) 2024-04-25T19:46:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.723+0000 7fd7fffff640 1 -- 172.21.15.29:0/224938414 learned_addr learned my addr 172.21.15.29:0/224938414 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:46:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.723+0000 7fd7feffd640 1 --2- 172.21.15.29:0/224938414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8000aceb0 0x7fd800144810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:04.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.723+0000 7fd7fffff640 1 -- 172.21.15.29:0/224938414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8000aceb0 msgr2=0x7fd800144810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:04.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.723+0000 7fd7fffff640 1 --2- 172.21.15.29:0/224938414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8000aceb0 0x7fd800144810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.723+0000 7fd7fffff640 1 -- 172.21.15.29:0/224938414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8000acaa0 msgr2=0x7fd800139fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:04.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.723+0000 7fd7fffff640 1 --2- 172.21.15.29:0/224938414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8000acaa0 0x7fd800139fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.723+0000 7fd7fffff640 1 -- 172.21.15.29:0/224938414 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd800143560 con 0x7fd800143e30 2024-04-25T19:46:04.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.723+0000 7fd7feffd640 1 --2- 172.21.15.29:0/224938414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8000aceb0 0x7fd800144810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:46:04.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.724+0000 7fd7fffff640 1 --2- 172.21.15.29:0/224938414 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd800143e30 0x7fd80009b1e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8036040 tx=0x7fd7f8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:04.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.724+0000 7fd7e7fff640 1 -- 172.21.15.29:0/224938414 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f8043020 con 0x7fd800143e30 2024-04-25T19:46:04.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.724+0000 7fd7e7fff640 1 -- 172.21.15.29:0/224938414 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fd7f803e450 con 0x7fd800143e30 2024-04-25T19:46:04.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.724+0000 7fd80611b640 1 -- 172.21.15.29:0/224938414 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd800144d50 con 0x7fd800143e30 2024-04-25T19:46:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.724+0000 7fd7e7fff640 1 -- 172.21.15.29:0/224938414 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f803e840 con 0x7fd800143e30 2024-04-25T19:46:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.724+0000 7fd80611b640 1 -- 172.21.15.29:0/224938414 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8000a5300 con 0x7fd800143e30 2024-04-25T19:46:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.725+0000 7fd80611b640 1 -- 172.21.15.29:0/224938414 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd800144f80 con 0x7fd800143e30 2024-04-25T19:46:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.726+0000 7fd7e7fff640 1 -- 172.21.15.29:0/224938414 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 14) ==== 144697+0+0 (secure 0 0 0) 0x7fd7f8046020 con 0x7fd800143e30 2024-04-25T19:46:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.726+0000 7fd7e7fff640 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-25T19:46:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.727+0000 7fd7e7fff640 4 mgrc handle_mgr_map Got map version 14 2024-04-25T19:46:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.727+0000 7fd7e7fff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] 2024-04-25T19:46:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.727+0000 7fd7e7fff640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] 2024-04-25T19:46:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.727+0000 7fd7e7fff640 1 --2- 172.21.15.29:0/224938414 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7fd7ec0ac3b0 0x7fd7ec0ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.727+0000 7fd7e7fff640 1 -- 172.21.15.29:0/224938414 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(31..31 src has 1..31) ==== 5198+0+0 (secure 0 0 0) 0x7fd7f80fc140 con 0x7fd800143e30 2024-04-25T19:46:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.727+0000 7fd7ff7fe640 1 --2- 172.21.15.29:0/224938414 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7fd7ec0ac3b0 0x7fd7ec0ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.728+0000 7fd7ff7fe640 1 --2- 172.21.15.29:0/224938414 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7fd7ec0ac3b0 0x7fd7ec0ae8c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd7f00079a0 tx=0x7fd7f0008040 comp rx=0 tx=0).ready entity=mgr.4595 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.730+0000 7fd7e7fff640 1 -- 172.21.15.29:0/224938414 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd800144f80 con 0x7fd800143e30 2024-04-25T19:46:04.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.867+0000 7fd7e7fff640 1 -- 172.21.15.29:0/224938414 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mgrmap(e 15) ==== 144905+0+0 (secure 0 0 0) 0x7fd7f80c27c0 con 0x7fd800143e30 2024-04-25T19:46:04.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.867+0000 7fd7e7fff640 20 mgrc handle_mgr_map mgrmap(e 15) 2024-04-25T19:46:04.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.868+0000 7fd7e7fff640 4 mgrc handle_mgr_map Got map version 15 2024-04-25T19:46:04.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.868+0000 7fd7e7fff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] 2024-04-25T19:46:04.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.920+0000 7fd80611b640 1 -- 172.21.15.29:0/224938414 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd8000507b0 con 0x7fd800143e30 2024-04-25T19:46:04.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.921+0000 7fd7e7fff640 1 -- 172.21.15.29:0/224938414 <== mon.2 v2:172.21.15.29:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) ==== 74+0+14644 (secure 0 0 0) 0x7fd7f804e020 con 0x7fd800143e30 2024-04-25T19:46:04.922 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-25T19:46:04.922 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"fsid":"1b3c39b5-e787-409e-b0bd-f4f491c89d72","created":"2024-04-25T19:43:47.854697+0000","modified":"2024-04-25T19:46:02.778051+0000","last_up_change":"2024-04-25T19:43:57.853237+0000","last_in_change":"2024-04-25T19:43:49.022776+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:43:58.096700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T19:44:03.617011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:44:07.890493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2024-04-25T19:44:08.355570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.190000057220459,"score_stable":1.190000057220459,"optimal_score":1,"raw_score_acting":1.190000057220459,"raw_score_stable":1.190000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d805223-ee84-494f-bd31-59091906aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6801","nonce":695704147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6803","nonce":695704147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6807","nonce":695704147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":695704147},{"type":"v1","addr":"172.21.15.29:6805","nonce":695704147}]},"public_addr":"172.21.15.29:6801/695704147","cluster_addr":"172.21.15.29:6803/695704147","heartbeat_back_addr":"172.21.15.29:6807/695704147","heartbeat_front_addr":"172.21.15.29:6805/695704147","state":["exists","up"]},{"osd":1,"uuid":"607c5038-38af-440f-8371-e71758549920","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6814","nonce":3135821452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6817","nonce":3135821452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6823","nonce":3135821452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":3135821452},{"type":"v1","addr":"172.21.15.102:6821","nonce":3135821452}]},"public_addr":"172.21.15.102:6814/3135821452","cluster_addr":"172.21.15.102:6817/3135821452","heartbeat_back_addr":"172.21.15.102:6823/3135821452","heartbeat_front_addr":"172.21.15.102:6821/3135821452","state":["exists","up"]},{"osd":2,"uuid":"6859ae76-54bb-4c55-b0dc-5e2735c0d31c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6801","nonce":2071844144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6803","nonce":2071844144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6811","nonce":2071844144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":2071844144},{"type":"v1","addr":"172.21.15.102:6806","nonce":2071844144}]},"public_addr":"172.21.15.102:6801/2071844144","cluster_addr":"172.21.15.102:6803/2071844144","heartbeat_back_addr":"172.21.15.102:6811/2071844144","heartbeat_front_addr":"172.21.15.102:6806/2071844144","state":["exists","up"]},{"osd":3,"uuid":"a98b7b88-5206-4f22-84a6-78212e0d5f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6805","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6807","nonce":339944984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6810","nonce":339944984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6820","nonce":339944984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":339944984},{"type":"v1","addr":"172.21.15.102:6815","nonce":339944984}]},"public_addr":"172.21.15.102:6807/339944984","cluster_addr":"172.21.15.102:6810/339944984","heartbeat_back_addr":"172.21.15.102:6820/339944984","heartbeat_front_addr":"172.21.15.102:6815/339944984","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.121529+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.201155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.264387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:43:54.198713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2","mappings":[{"from":3,"to":0}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.12","mappings":[{"from":3,"to":2}]},{"pgid":"4.1d","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/3938108138":"2024-04-26T19:46:02.778000+0000","172.21.15.102:0/1318055012":"2024-04-26T19:45:55.002117+0000","172.21.15.102:6824/1500823528":"2024-04-26T19:46:02.778000+0000","172.21.15.102:0/1813070992":"2024-04-26T19:45:55.002117+0000","172.21.15.102:6824/1439704316":"2024-04-26T19:45:55.002117+0000","172.21.15.102:0/694371794":"2024-04-26T19:46:02.778000+0000","172.21.15.102:0/2274001855":"2024-04-26T19:45:55.002117+0000","172.21.15.102:0/2242392158":"2024-04-26T19:46:02.778000+0000","172.21.15.102:0/1603274171":"2024-04-26T19:46:02.778000+0000","172.21.15.102:0/4278965720":"2024-04-26T19:45:55.002117+0000","172.21.15.102:6825/1500823528":"2024-04-26T19:46:02.778000+0000","172.21.15.102:6825/1439704316":"2024-04-26T19:45:55.002117+0000","172.21.15.102:0/1957717782":"2024-04-26T19:46:02.778000+0000","172.21.15.102:0/4105559723":"2024-04-26T19:46:02.778000+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:46:04.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.923+0000 7fd80611b640 10 mgrc shutdown 2024-04-25T19:46:04.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.923+0000 7fd80611b640 1 -- 172.21.15.29:0/224938414 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7fd7ec0ac3b0 msgr2=0x7fd7ec0ae8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:04.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.923+0000 7fd80611b640 1 --2- 172.21.15.29:0/224938414 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7fd7ec0ac3b0 0x7fd7ec0ae8c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd7f00079a0 tx=0x7fd7f0008040 comp rx=0 tx=0).stop 2024-04-25T19:46:04.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.923+0000 7fd80611b640 1 -- 172.21.15.29:0/224938414 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd800143e30 msgr2=0x7fd80009b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:04.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.923+0000 7fd80611b640 1 --2- 172.21.15.29:0/224938414 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd800143e30 0x7fd80009b1e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8036040 tx=0x7fd7f8004a40 comp rx=0 tx=0).stop 2024-04-25T19:46:04.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.923+0000 7fd80611b640 1 -- 172.21.15.29:0/224938414 shutdown_connections 2024-04-25T19:46:04.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.923+0000 7fd80611b640 1 --2- 172.21.15.29:0/224938414 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7fd7ec0ac3b0 0x7fd7ec0ae8c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.923+0000 7fd80611b640 1 --2- 172.21.15.29:0/224938414 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd800143e30 0x7fd80009b1e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.923+0000 7fd80611b640 1 --2- 172.21.15.29:0/224938414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8000aceb0 0x7fd800144810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.923+0000 7fd80611b640 1 --2- 172.21.15.29:0/224938414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8000acaa0 0x7fd800139fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:04.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.923+0000 7fd80611b640 1 -- 172.21.15.29:0/224938414 >> 172.21.15.29:0/224938414 conn(0x7fd8000aafc0 msgr2=0x7fd8000a7550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:04.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.923+0000 7fd80611b640 1 -- 172.21.15.29:0/224938414 shutdown_connections 2024-04-25T19:46:04.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:04.923+0000 7fd80611b640 1 -- 172.21.15.29:0/224938414 wait complete. 2024-04-25T19:46:04.936 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T19:46:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.046+0000 7f654ffef640 1 Processor -- start 2024-04-25T19:46:05.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.046+0000 7f654ffef640 1 -- start start 2024-04-25T19:46:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.047+0000 7f654ffef640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6548144240 0x7f6548146c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.047+0000 7f654ffef640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6548144650 con 0x7f6548143e30 2024-04-25T19:46:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.047+0000 7f654ffef640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65480a5a90 con 0x7f6548147150 2024-04-25T19:46:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.047+0000 7f654ffef640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f6548147560 con 0x7f6548144240 2024-04-25T19:46:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.048+0000 7f654d58a640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6548144240 0x7f6548146c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:05.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.048+0000 7f654dd8b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6548143e30 0x7f65480acaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43270/0 (socket says 172.21.15.29:43270) 2024-04-25T19:46:05.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.048+0000 7f654dd8b640 1 -- 172.21.15.29:0/393667260 learned_addr learned my addr 172.21.15.29:0/393667260 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:46:05.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.048+0000 7f654cd89640 1 -- 172.21.15.29:0/393667260 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 316760205 0 0) 0x7f6540001110 con 0x7f6548143e30 2024-04-25T19:46:05.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.048+0000 7f654d58a640 1 -- 172.21.15.29:0/393667260 >> v1:172.21.15.102:6789/0 conn(0x7f6548147150 legacy=0x7f6548151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:05.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.048+0000 7f654d58a640 1 -- 172.21.15.29:0/393667260 >> v1:172.21.15.29:6789/0 conn(0x7f6548143e30 legacy=0x7f65480acaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:05.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.048+0000 7f654d58a640 1 -- 172.21.15.29:0/393667260 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6548147790 con 0x7f6548144240 2024-04-25T19:46:05.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.049+0000 7f654d58a640 1 --2- 172.21.15.29:0/393667260 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6548144240 0x7f6548146c10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6538009800 tx=0x7f653802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=501d16608add322b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:05.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.049+0000 7f654cd89640 1 -- 172.21.15.29:0/393667260 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6538035020 con 0x7f6548144240 2024-04-25T19:46:05.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.049+0000 7f654cd89640 1 -- 172.21.15.29:0/393667260 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f6538031890 con 0x7f6548144240 2024-04-25T19:46:05.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.049+0000 7f654cd89640 1 -- 172.21.15.29:0/393667260 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6538031c80 con 0x7f6548144240 2024-04-25T19:46:05.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.049+0000 7f654ffef640 1 -- 172.21.15.29:0/393667260 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6548144240 msgr2=0x7f6548146c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:05.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.049+0000 7f654ffef640 1 --2- 172.21.15.29:0/393667260 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6548144240 0x7f6548146c10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6538009800 tx=0x7f653802b930 comp rx=0 tx=0).stop 2024-04-25T19:46:05.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.050+0000 7f654ffef640 1 -- 172.21.15.29:0/393667260 shutdown_connections 2024-04-25T19:46:05.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.050+0000 7f654ffef640 1 --2- 172.21.15.29:0/393667260 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6548144240 0x7f6548146c10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:05.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.050+0000 7f654ffef640 1 -- 172.21.15.29:0/393667260 >> 172.21.15.29:0/393667260 conn(0x7f65480aafc0 msgr2=0x7f65480ab3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:05.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.050+0000 7f654ffef640 1 -- 172.21.15.29:0/393667260 shutdown_connections 2024-04-25T19:46:05.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.050+0000 7f654ffef640 1 -- 172.21.15.29:0/393667260 wait complete. 2024-04-25T19:46:05.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.051+0000 7f654ffef640 1 Processor -- start 2024-04-25T19:46:05.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.051+0000 7f654ffef640 1 -- start start 2024-04-25T19:46:05.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.052+0000 7f654ffef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6548143e30 0x7f654814a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:05.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.052+0000 7f654ffef640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6548144240 0x7f6548139710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:05.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.052+0000 7f654ffef640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6548147150 0x7f654814b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:05.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.052+0000 7f654ffef640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6548152cb0 con 0x7f6548143e30 2024-04-25T19:46:05.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.052+0000 7f654ffef640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6548147a80 con 0x7f6548147150 2024-04-25T19:46:05.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.052+0000 7f654ffef640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f6548152e30 con 0x7f6548144240 2024-04-25T19:46:05.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.052+0000 7f654d58a640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6548144240 0x7f6548139710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:05.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.052+0000 7f654dd8b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6548143e30 0x7f654814a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:05.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.052+0000 7f654d58a640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6548144240 0x7f6548139710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:50054/0 (socket says 172.21.15.29:50054) 2024-04-25T19:46:05.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.052+0000 7f654d58a640 1 -- 172.21.15.29:0/2210808695 learned_addr learned my addr 172.21.15.29:0/2210808695 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:46:05.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.052+0000 7f654dd8b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6548143e30 0x7f654814a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51274/0 (socket says 172.21.15.29:51274) 2024-04-25T19:46:05.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.052+0000 7f654e58c640 1 --2- 172.21.15.29:0/2210808695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6548147150 0x7f654814b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:05.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.052+0000 7f654d58a640 1 -- 172.21.15.29:0/2210808695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6548147150 msgr2=0x7f654814b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:05.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.052+0000 7f654d58a640 1 --2- 172.21.15.29:0/2210808695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6548147150 0x7f654814b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:05.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.053+0000 7f654d58a640 1 -- 172.21.15.29:0/2210808695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6548143e30 msgr2=0x7f654814a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:05.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.053+0000 7f654d58a640 1 --2- 172.21.15.29:0/2210808695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6548143e30 0x7f654814a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.053+0000 7f654d58a640 1 -- 172.21.15.29:0/2210808695 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f654814abf0 con 0x7f6548144240 2024-04-25T19:46:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.053+0000 7f654dd8b640 1 --2- 172.21.15.29:0/2210808695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6548143e30 0x7f654814a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:46:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.053+0000 7f654d58a640 1 --2- 172.21.15.29:0/2210808695 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6548144240 0x7f6548139710 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6538009720 tx=0x7f65380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.053+0000 7f653e7fc640 1 -- 172.21.15.29:0/2210808695 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6538035040 con 0x7f6548144240 2024-04-25T19:46:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.053+0000 7f653e7fc640 1 -- 172.21.15.29:0/2210808695 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f6538004020 con 0x7f6548144240 2024-04-25T19:46:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.053+0000 7f654ffef640 1 -- 172.21.15.29:0/2210808695 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f654814e5f0 con 0x7f6548144240 2024-04-25T19:46:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.053+0000 7f653e7fc640 1 -- 172.21.15.29:0/2210808695 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6538004430 con 0x7f6548144240 2024-04-25T19:46:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.053+0000 7f654ffef640 1 -- 172.21.15.29:0/2210808695 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6548141430 con 0x7f6548144240 2024-04-25T19:46:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.054+0000 7f654ffef640 1 -- 172.21.15.29:0/2210808695 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f654813a450 con 0x7f6548144240 2024-04-25T19:46:05.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.056+0000 7f653e7fc640 1 -- 172.21.15.29:0/2210808695 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 15) ==== 144905+0+0 (secure 0 0 0) 0x7f6538038020 con 0x7f6548144240 2024-04-25T19:46:05.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.056+0000 7f653e7fc640 20 mgrc handle_mgr_map mgrmap(e 15) 2024-04-25T19:46:05.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.057+0000 7f653e7fc640 4 mgrc handle_mgr_map Got map version 15 2024-04-25T19:46:05.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.057+0000 7f653e7fc640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] 2024-04-25T19:46:05.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.057+0000 7f653e7fc640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] 2024-04-25T19:46:05.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.057+0000 7f653e7fc640 1 --2- 172.21.15.29:0/2210808695 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f652c0ac5c0 0x7f652c0aead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:05.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.057+0000 7f653e7fc640 1 -- 172.21.15.29:0/2210808695 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(31..31 src has 1..31) ==== 5198+0+0 (secure 0 0 0) 0x7f65380fbfe0 con 0x7f6548144240 2024-04-25T19:46:05.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.057+0000 7f654dd8b640 1 --2- 172.21.15.29:0/2210808695 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f652c0ac5c0 0x7f652c0aead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:05.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.058+0000 7f654dd8b640 1 --2- 172.21.15.29:0/2210808695 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f652c0ac5c0 0x7f652c0aead0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6540000f60 tx=0x7f6540007e40 comp rx=0 tx=0).ready entity=mgr.4595 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:05.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.060+0000 7f653e7fc640 1 -- 172.21.15.29:0/2210808695 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f653802d040 con 0x7f6548144240 2024-04-25T19:46:05.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.282+0000 7f654ffef640 1 -- 172.21.15.29:0/2210808695 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f65480507b0 con 0x7f6548144240 2024-04-25T19:46:05.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.289+0000 7f653e7fc640 1 -- 172.21.15.29:0/2210808695 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) ==== 161+0+0 (secure 0 0 0) 0x7f65380c61b0 con 0x7f6548144240 2024-04-25T19:46:05.290 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T19:46:05.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.291+0000 7f654ffef640 10 mgrc shutdown 2024-04-25T19:46:05.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.291+0000 7f654ffef640 1 -- 172.21.15.29:0/2210808695 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f652c0ac5c0 msgr2=0x7f652c0aead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:05.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.291+0000 7f654ffef640 1 --2- 172.21.15.29:0/2210808695 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f652c0ac5c0 0x7f652c0aead0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6540000f60 tx=0x7f6540007e40 comp rx=0 tx=0).stop 2024-04-25T19:46:05.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.291+0000 7f654ffef640 1 -- 172.21.15.29:0/2210808695 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6548144240 msgr2=0x7f6548139710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:05.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.292+0000 7f654ffef640 1 --2- 172.21.15.29:0/2210808695 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6548144240 0x7f6548139710 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6538009720 tx=0x7f65380046c0 comp rx=0 tx=0).stop 2024-04-25T19:46:05.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.292+0000 7f654ffef640 1 -- 172.21.15.29:0/2210808695 shutdown_connections 2024-04-25T19:46:05.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.292+0000 7f654ffef640 1 --2- 172.21.15.29:0/2210808695 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f652c0ac5c0 0x7f652c0aead0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:05.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.292+0000 7f654ffef640 1 --2- 172.21.15.29:0/2210808695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6548147150 0x7f654814b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:05.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.292+0000 7f654ffef640 1 --2- 172.21.15.29:0/2210808695 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6548144240 0x7f6548139710 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:05.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.292+0000 7f654ffef640 1 --2- 172.21.15.29:0/2210808695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6548143e30 0x7f654814a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:05.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.292+0000 7f654ffef640 1 -- 172.21.15.29:0/2210808695 >> 172.21.15.29:0/2210808695 conn(0x7f65480aafc0 msgr2=0x7f65480a64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:05.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.292+0000 7f654ffef640 1 -- 172.21.15.29:0/2210808695 shutdown_connections 2024-04-25T19:46:05.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.292+0000 7f654ffef640 1 -- 172.21.15.29:0/2210808695 wait complete. 2024-04-25T19:46:05.305 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T19:46:05.341 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2024-04-25T19:46:05.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.417+0000 7f34fd698640 1 Processor -- start 2024-04-25T19:46:05.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.417+0000 7f34fd698640 1 -- start start 2024-04-25T19:46:05.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.418+0000 7f34fd698640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f34f806d490 0x7f34f80ac6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:05.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.418+0000 7f34fd698640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f34f814f300 0x7f34f8151720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:05.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.418+0000 7f34fd698640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34f806d8a0 con 0x7f34f80ad130 2024-04-25T19:46:05.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.418+0000 7f34fd698640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f34f8146960 con 0x7f34f814f300 2024-04-25T19:46:05.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.418+0000 7f34fd698640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f34f80ad880 con 0x7f34f806d490 2024-04-25T19:46:05.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.418+0000 7f34f77fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f34f814f300 0x7f34f8151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:05.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.418+0000 7f34f67fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f34f80ad130 0x7f34f814a870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43288/0 (socket says 172.21.15.29:43288) 2024-04-25T19:46:05.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.418+0000 7f34f77fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f34f814f300 0x7f34f8151720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51316/0 (socket says 172.21.15.29:51316) 2024-04-25T19:46:05.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.418+0000 7f34f67fc640 1 -- 172.21.15.29:0/664085141 learned_addr learned my addr 172.21.15.29:0/664085141 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:46:05.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.419+0000 7f34f6ffd640 1 --2- 172.21.15.29:0/664085141 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f34f806d490 0x7f34f80ac6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:05.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.419+0000 7f34f77fe640 1 -- 172.21.15.29:0/664085141 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f34f806d490 msgr2=0x7f34f80ac6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:46:05.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.419+0000 7f34f77fe640 1 --2- 172.21.15.29:0/664085141 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f34f806d490 0x7f34f80ac6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:05.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.419+0000 7f34f77fe640 1 -- 172.21.15.29:0/664085141 >> v1:172.21.15.29:6789/0 conn(0x7f34f80ad130 legacy=0x7f34f814a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:05.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.419+0000 7f34f5ffb640 1 -- 172.21.15.29:0/664085141 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4052065800 0 0) 0x7f34ec001110 con 0x7f34f80ad130 2024-04-25T19:46:05.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.419+0000 7f34f77fe640 1 -- 172.21.15.29:0/664085141 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f8149fe0 con 0x7f34f814f300 2024-04-25T19:46:05.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.419+0000 7f34f6ffd640 1 --2- 172.21.15.29:0/664085141 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f34f806d490 0x7f34f80ac6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:46:05.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.420+0000 7f34f77fe640 1 --2- 172.21.15.29:0/664085141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34f814f300 0x7f34f8151720 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f34f00097a0 tx=0x7f34f002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d3eab7505dace78 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:05.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.420+0000 7f34f5ffb640 1 -- 172.21.15.29:0/664085141 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f00315a0 con 0x7f34f814f300 2024-04-25T19:46:05.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.420+0000 7f34f5ffb640 1 -- 172.21.15.29:0/664085141 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f34f0037070 con 0x7f34f814f300 2024-04-25T19:46:05.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.420+0000 7f34f5ffb640 1 -- 172.21.15.29:0/664085141 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f0031980 con 0x7f34f814f300 2024-04-25T19:46:05.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.420+0000 7f34fd698640 1 -- 172.21.15.29:0/664085141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34f814f300 msgr2=0x7f34f8151720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:05.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.420+0000 7f34fd698640 1 --2- 172.21.15.29:0/664085141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34f814f300 0x7f34f8151720 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f34f00097a0 tx=0x7f34f002b840 comp rx=0 tx=0).stop 2024-04-25T19:46:05.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.420+0000 7f34fd698640 1 -- 172.21.15.29:0/664085141 shutdown_connections 2024-04-25T19:46:05.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.420+0000 7f34fd698640 1 --2- 172.21.15.29:0/664085141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34f814f300 0x7f34f8151720 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:05.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.420+0000 7f34fd698640 1 --2- 172.21.15.29:0/664085141 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f34f806d490 0x7f34f80ac6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:05.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.420+0000 7f34fd698640 1 -- 172.21.15.29:0/664085141 >> 172.21.15.29:0/664085141 conn(0x7f34f806c790 msgr2=0x7f34f806cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:05.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.421+0000 7f34fd698640 1 -- 172.21.15.29:0/664085141 shutdown_connections 2024-04-25T19:46:05.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.421+0000 7f34fd698640 1 -- 172.21.15.29:0/664085141 wait complete. 2024-04-25T19:46:05.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.422+0000 7f34fd698640 1 Processor -- start 2024-04-25T19:46:05.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.422+0000 7f34fd698640 1 -- start start 2024-04-25T19:46:05.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.423+0000 7f34fd698640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34f806d490 0x7f34f80baf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:05.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.423+0000 7f34fd698640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f34f80ad130 0x7f34f814d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:05.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.423+0000 7f34fd698640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34f814f300 0x7f34f8145270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:05.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.423+0000 7f34fd698640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f34f80acd60 con 0x7f34f806d490 2024-04-25T19:46:05.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.423+0000 7f34fd698640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f34f80ad540 con 0x7f34f814f300 2024-04-25T19:46:05.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.423+0000 7f34fd698640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f34f80acee0 con 0x7f34f80ad130 2024-04-25T19:46:05.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.423+0000 7f34f6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34f806d490 0x7f34f80baf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:05.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.423+0000 7f34f67fc640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f34f80ad130 0x7f34f814d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:05.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.423+0000 7f34f6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34f806d490 0x7f34f80baf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51322/0 (socket says 172.21.15.29:51322) 2024-04-25T19:46:05.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.423+0000 7f34f6ffd640 1 -- 172.21.15.29:0/2902892298 learned_addr learned my addr 172.21.15.29:0/2902892298 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:46:05.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.423+0000 7f34f77fe640 1 --2- 172.21.15.29:0/2902892298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34f814f300 0x7f34f8145270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:05.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.424+0000 7f34f67fc640 1 -- 172.21.15.29:0/2902892298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34f814f300 msgr2=0x7f34f8145270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:05.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.424+0000 7f34f67fc640 1 --2- 172.21.15.29:0/2902892298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34f814f300 0x7f34f8145270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:05.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.424+0000 7f34f67fc640 1 -- 172.21.15.29:0/2902892298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34f806d490 msgr2=0x7f34f80baf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:05.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.424+0000 7f34f67fc640 1 --2- 172.21.15.29:0/2902892298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34f806d490 0x7f34f80baf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:05.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.424+0000 7f34f67fc640 1 -- 172.21.15.29:0/2902892298 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f814d120 con 0x7f34f80ad130 2024-04-25T19:46:05.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.424+0000 7f34f6ffd640 1 --2- 172.21.15.29:0/2902892298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34f806d490 0x7f34f80baf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:46:05.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.424+0000 7f34f67fc640 1 --2- 172.21.15.29:0/2902892298 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f34f80ad130 0x7f34f814d640 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f34ec00c3e0 tx=0x7f34ec00c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:05.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.424+0000 7f34df7fe640 1 -- 172.21.15.29:0/2902892298 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ec017070 con 0x7f34f80ad130 2024-04-25T19:46:05.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.424+0000 7f34df7fe640 1 -- 172.21.15.29:0/2902892298 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f34ec005d00 con 0x7f34f80ad130 2024-04-25T19:46:05.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.424+0000 7f34fd698640 1 -- 172.21.15.29:0/2902892298 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34f81438e0 con 0x7f34f80ad130 2024-04-25T19:46:05.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.424+0000 7f34df7fe640 1 -- 172.21.15.29:0/2902892298 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ec007330 con 0x7f34f80ad130 2024-04-25T19:46:05.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.424+0000 7f34fd698640 1 -- 172.21.15.29:0/2902892298 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f34f813ec90 con 0x7f34f80ad130 2024-04-25T19:46:05.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.425+0000 7f34fd698640 1 -- 172.21.15.29:0/2902892298 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34f8050740 con 0x7f34f80ad130 2024-04-25T19:46:05.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.427+0000 7f34df7fe640 1 -- 172.21.15.29:0/2902892298 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 15) ==== 144905+0+0 (secure 0 0 0) 0x7f34ec01a070 con 0x7f34f80ad130 2024-04-25T19:46:05.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.427+0000 7f34df7fe640 20 mgrc handle_mgr_map mgrmap(e 15) 2024-04-25T19:46:05.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.428+0000 7f34df7fe640 4 mgrc handle_mgr_map Got map version 15 2024-04-25T19:46:05.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.428+0000 7f34df7fe640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] 2024-04-25T19:46:05.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.428+0000 7f34df7fe640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] 2024-04-25T19:46:05.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.428+0000 7f34df7fe640 1 --2- 172.21.15.29:0/2902892298 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f34e40ac5f0 0x7f34e40aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:05.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.428+0000 7f34df7fe640 1 -- 172.21.15.29:0/2902892298 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(32..32 src has 1..32) ==== 5284+0+0 (secure 0 0 0) 0x7f34ec0d8ef0 con 0x7f34f80ad130 2024-04-25T19:46:05.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.428+0000 7f34f6ffd640 1 --2- 172.21.15.29:0/2902892298 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f34e40ac5f0 0x7f34e40aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:05.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.429+0000 7f34f6ffd640 1 --2- 172.21.15.29:0/2902892298 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f34e40ac5f0 0x7f34e40aeb00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f34e8009bd0 tx=0x7f34e800b040 comp rx=0 tx=0).ready entity=mgr.4595 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:05.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.431+0000 7f34df7fe640 1 -- 172.21.15.29:0/2902892298 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f34ec021020 con 0x7f34f80ad130 2024-04-25T19:46:05.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:05.659+0000 7f34fd698640 1 -- 172.21.15.29:0/2902892298 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f34f80a8830 con 0x7f34f80ad130 2024-04-25T19:46:06.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.290+0000 7f34df7fe640 1 -- 172.21.15.29:0/2902892298 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) ==== 104+0+0 (secure 0 0 0) 0x7f34f813ec90 con 0x7f34f80ad130 2024-04-25T19:46:06.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.292+0000 7f34fd698640 10 mgrc shutdown 2024-04-25T19:46:06.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.292+0000 7f34fd698640 1 -- 172.21.15.29:0/2902892298 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f34e40ac5f0 msgr2=0x7f34e40aeb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:06.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.292+0000 7f34fd698640 1 --2- 172.21.15.29:0/2902892298 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f34e40ac5f0 0x7f34e40aeb00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f34e8009bd0 tx=0x7f34e800b040 comp rx=0 tx=0).stop 2024-04-25T19:46:06.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.292+0000 7f34fd698640 1 -- 172.21.15.29:0/2902892298 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f34f80ad130 msgr2=0x7f34f814d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:06.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.292+0000 7f34fd698640 1 --2- 172.21.15.29:0/2902892298 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f34f80ad130 0x7f34f814d640 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f34ec00c3e0 tx=0x7f34ec00c8a0 comp rx=0 tx=0).stop 2024-04-25T19:46:06.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.292+0000 7f34fd698640 1 -- 172.21.15.29:0/2902892298 shutdown_connections 2024-04-25T19:46:06.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.292+0000 7f34fd698640 1 --2- 172.21.15.29:0/2902892298 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f34e40ac5f0 0x7f34e40aeb00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:06.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.292+0000 7f34fd698640 1 --2- 172.21.15.29:0/2902892298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34f814f300 0x7f34f8145270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:06.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.292+0000 7f34fd698640 1 --2- 172.21.15.29:0/2902892298 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f34f80ad130 0x7f34f814d640 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:06.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.292+0000 7f34fd698640 1 --2- 172.21.15.29:0/2902892298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34f806d490 0x7f34f80baf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:06.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.292+0000 7f34fd698640 1 -- 172.21.15.29:0/2902892298 >> 172.21.15.29:0/2902892298 conn(0x7f34f806c790 msgr2=0x7f34f8150140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:06.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.292+0000 7f34fd698640 1 -- 172.21.15.29:0/2902892298 shutdown_connections 2024-04-25T19:46:06.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.293+0000 7f34fd698640 1 -- 172.21.15.29:0/2902892298 wait complete. 2024-04-25T19:46:06.307 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T19:46:06.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.471+0000 7f1f3996d640 1 Processor -- start 2024-04-25T19:46:06.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.471+0000 7f1f3996d640 1 -- start start 2024-04-25T19:46:06.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.472+0000 7f1f3996d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1f340a55d0 0x7f1f34146140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:06.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.472+0000 7f1f3996d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f340ad300 con 0x7f1f34146680 2024-04-25T19:46:06.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.472+0000 7f1f3996d640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f340ad5c0 con 0x7f1f340ae260 2024-04-25T19:46:06.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.472+0000 7f1f3996d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f1f340a5ff0 con 0x7f1f340a55d0 2024-04-25T19:46:06.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.472+0000 7f1f327fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1f340a55d0 0x7f1f34146140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:06.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.472+0000 7f1f327fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1f340a55d0 0x7f1f34146140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51324/0 (socket says 172.21.15.29:51324) 2024-04-25T19:46:06.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.472+0000 7f1f327fc640 1 -- 172.21.15.29:0/1852630512 learned_addr learned my addr 172.21.15.29:0/1852630512 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:46:06.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.472+0000 7f1f337fe640 1 --1- 172.21.15.29:0/1852630512 >> v1:172.21.15.29:6789/0 conn(0x7f1f34146680 0x7f1f341513b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43292/0 (socket says 172.21.15.29:43292) 2024-04-25T19:46:06.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.473+0000 7f1f327fc640 1 -- 172.21.15.29:0/1852630512 >> v1:172.21.15.102:6789/0 conn(0x7f1f340ae260 legacy=0x7f1f340acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:06.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.473+0000 7f1f327fc640 1 -- 172.21.15.29:0/1852630512 >> v1:172.21.15.29:6789/0 conn(0x7f1f34146680 legacy=0x7f1f341513b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:06.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.473+0000 7f1f327fc640 1 -- 172.21.15.29:0/1852630512 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f340aad10 con 0x7f1f340a55d0 2024-04-25T19:46:06.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.473+0000 7f1f31ffb640 1 -- 172.21.15.29:0/1852630512 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2080847133 0 0) 0x7f1f340ad300 con 0x7f1f34146680 2024-04-25T19:46:06.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.473+0000 7f1f327fc640 1 --2- 172.21.15.29:0/1852630512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f340a55d0 0x7f1f34146140 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1f28009870 tx=0x7f1f2802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d9b6a9e967c8efc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:06.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.474+0000 7f1f31ffb640 1 -- 172.21.15.29:0/1852630512 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f2802f840 con 0x7f1f340a55d0 2024-04-25T19:46:06.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.474+0000 7f1f31ffb640 1 -- 172.21.15.29:0/1852630512 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f1f2802f9e0 con 0x7f1f340a55d0 2024-04-25T19:46:06.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.474+0000 7f1f31ffb640 1 -- 172.21.15.29:0/1852630512 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f2802fdd0 con 0x7f1f340a55d0 2024-04-25T19:46:06.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.474+0000 7f1f3996d640 1 -- 172.21.15.29:0/1852630512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f340a55d0 msgr2=0x7f1f34146140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:06.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.474+0000 7f1f3996d640 1 --2- 172.21.15.29:0/1852630512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f340a55d0 0x7f1f34146140 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1f28009870 tx=0x7f1f2802ba20 comp rx=0 tx=0).stop 2024-04-25T19:46:06.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.474+0000 7f1f3996d640 1 -- 172.21.15.29:0/1852630512 shutdown_connections 2024-04-25T19:46:06.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.474+0000 7f1f3996d640 1 --2- 172.21.15.29:0/1852630512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f340a55d0 0x7f1f34146140 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:06.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.474+0000 7f1f3996d640 1 -- 172.21.15.29:0/1852630512 >> 172.21.15.29:0/1852630512 conn(0x7f1f340ab3f0 msgr2=0x7f1f340ab800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:06.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.474+0000 7f1f3996d640 1 -- 172.21.15.29:0/1852630512 shutdown_connections 2024-04-25T19:46:06.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.474+0000 7f1f3996d640 1 -- 172.21.15.29:0/1852630512 wait complete. 2024-04-25T19:46:06.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.476+0000 7f1f3996d640 1 Processor -- start 2024-04-25T19:46:06.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.476+0000 7f1f3996d640 1 -- start start 2024-04-25T19:46:06.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.477+0000 7f1f3996d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f340a55d0 0x7f1f3414a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:06.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.477+0000 7f1f3996d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f340ae260 0x7f1f34141a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:06.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.477+0000 7f1f3996d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1f34146680 0x7f1f3413d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:06.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.477+0000 7f1f3996d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1f34152c50 con 0x7f1f340ae260 2024-04-25T19:46:06.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.477+0000 7f1f3996d640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1f34152ad0 con 0x7f1f340a55d0 2024-04-25T19:46:06.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.477+0000 7f1f3996d640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f1f34152dd0 con 0x7f1f34146680 2024-04-25T19:46:06.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.477+0000 7f1f327fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f340ae260 0x7f1f34141a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:06.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.477+0000 7f1f327fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f340ae260 0x7f1f34141a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51332/0 (socket says 172.21.15.29:51332) 2024-04-25T19:46:06.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.477+0000 7f1f327fc640 1 -- 172.21.15.29:0/2373485226 learned_addr learned my addr 172.21.15.29:0/2373485226 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:46:06.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.477+0000 7f1f337fe640 1 --2- 172.21.15.29:0/2373485226 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1f34146680 0x7f1f3413d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:06.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.477+0000 7f1f32ffd640 1 --2- 172.21.15.29:0/2373485226 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f340a55d0 0x7f1f3414a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:06.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.477+0000 7f1f327fc640 1 -- 172.21.15.29:0/2373485226 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1f34146680 msgr2=0x7f1f3413d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:06.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.477+0000 7f1f327fc640 1 --2- 172.21.15.29:0/2373485226 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1f34146680 0x7f1f3413d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:06.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.477+0000 7f1f327fc640 1 -- 172.21.15.29:0/2373485226 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f340a55d0 msgr2=0x7f1f3414a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:06.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.477+0000 7f1f327fc640 1 --2- 172.21.15.29:0/2373485226 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f340a55d0 0x7f1f3414a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:06.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.477+0000 7f1f327fc640 1 -- 172.21.15.29:0/2373485226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f3414a690 con 0x7f1f340ae260 2024-04-25T19:46:06.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.478+0000 7f1f327fc640 1 --2- 172.21.15.29:0/2373485226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f340ae260 0x7f1f34141a70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1f280090b0 tx=0x7f1f28004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:06.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.478+0000 7f1f1bfff640 1 -- 172.21.15.29:0/2373485226 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f28047070 con 0x7f1f340ae260 2024-04-25T19:46:06.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.478+0000 7f1f1bfff640 1 -- 172.21.15.29:0/2373485226 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f1f28004030 con 0x7f1f340ae260 2024-04-25T19:46:06.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.478+0000 7f1f3996d640 1 -- 172.21.15.29:0/2373485226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f3413d1a0 con 0x7f1f340ae260 2024-04-25T19:46:06.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.478+0000 7f1f1bfff640 1 -- 172.21.15.29:0/2373485226 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f28004440 con 0x7f1f340ae260 2024-04-25T19:46:06.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.478+0000 7f1f3996d640 1 -- 172.21.15.29:0/2373485226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f34139150 con 0x7f1f340ae260 2024-04-25T19:46:06.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.479+0000 7f1f3996d640 1 -- 172.21.15.29:0/2373485226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f340a6280 con 0x7f1f340ae260 2024-04-25T19:46:06.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.481+0000 7f1f1bfff640 1 -- 172.21.15.29:0/2373485226 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 144905+0+0 (secure 0 0 0) 0x7f1f2804a070 con 0x7f1f340ae260 2024-04-25T19:46:06.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.481+0000 7f1f1bfff640 20 mgrc handle_mgr_map mgrmap(e 15) 2024-04-25T19:46:06.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.482+0000 7f1f1bfff640 4 mgrc handle_mgr_map Got map version 15 2024-04-25T19:46:06.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.482+0000 7f1f1bfff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] 2024-04-25T19:46:06.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.482+0000 7f1f1bfff640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] 2024-04-25T19:46:06.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.482+0000 7f1f1bfff640 1 --2- 172.21.15.29:0/2373485226 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f1f200ac5c0 0x7f1f200aead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:06.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.482+0000 7f1f1bfff640 1 -- 172.21.15.29:0/2373485226 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5284+0+0 (secure 0 0 0) 0x7f1f280fbf00 con 0x7f1f340ae260 2024-04-25T19:46:06.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.482+0000 7f1f32ffd640 1 --2- 172.21.15.29:0/2373485226 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f1f200ac5c0 0x7f1f200aead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:06.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.483+0000 7f1f32ffd640 1 --2- 172.21.15.29:0/2373485226 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f1f200ac5c0 0x7f1f200aead0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1f24000f60 tx=0x7f1f24009040 comp rx=0 tx=0).ready entity=mgr.4595 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:06.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.485+0000 7f1f1bfff640 1 -- 172.21.15.29:0/2373485226 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1f340a6280 con 0x7f1f340ae260 2024-04-25T19:46:06.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:06.671+0000 7f1f3996d640 1 -- 172.21.15.29:0/2373485226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1f341393e0 con 0x7f1f340ae260 2024-04-25T19:46:07.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.292+0000 7f1f1bfff640 1 -- 172.21.15.29:0/2373485226 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v33) ==== 181+0+0 (secure 0 0 0) 0x7f1f28033030 con 0x7f1f340ae260 2024-04-25T19:46:07.294 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2024-04-25T19:46:07.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.298+0000 7f1f3996d640 10 mgrc shutdown 2024-04-25T19:46:07.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.298+0000 7f1f3996d640 1 -- 172.21.15.29:0/2373485226 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f1f200ac5c0 msgr2=0x7f1f200aead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:07.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.298+0000 7f1f3996d640 1 --2- 172.21.15.29:0/2373485226 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f1f200ac5c0 0x7f1f200aead0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1f24000f60 tx=0x7f1f24009040 comp rx=0 tx=0).stop 2024-04-25T19:46:07.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.298+0000 7f1f3996d640 1 -- 172.21.15.29:0/2373485226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f340ae260 msgr2=0x7f1f34141a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:07.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.300+0000 7f1f1bfff640 1 -- 172.21.15.29:0/2373485226 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 16) ==== 144961+0+0 (secure 0 0 0) 0x7f1f2802f8a0 con 0x7f1f340ae260 2024-04-25T19:46:07.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.300+0000 7f1f1bfff640 20 mgrc handle_mgr_map mgrmap(e 16) 2024-04-25T19:46:07.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.300+0000 7f1f3996d640 1 --2- 172.21.15.29:0/2373485226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f340ae260 0x7f1f34141a70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1f280090b0 tx=0x7f1f28004730 comp rx=0 tx=0).stop 2024-04-25T19:46:07.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.300+0000 7f1f3996d640 1 -- 172.21.15.29:0/2373485226 shutdown_connections 2024-04-25T19:46:07.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.300+0000 7f1f3996d640 1 --2- 172.21.15.29:0/2373485226 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f1f200ac5c0 0x7f1f200aead0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1f24000f60 tx=0x7f1f24009040 comp rx=0 tx=0).stop 2024-04-25T19:46:07.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.300+0000 7f1f3996d640 1 --2- 172.21.15.29:0/2373485226 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1f34146680 0x7f1f3413d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:07.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.300+0000 7f1f3996d640 1 --2- 172.21.15.29:0/2373485226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f340ae260 0x7f1f34141a70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:07.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.300+0000 7f1f3996d640 1 --2- 172.21.15.29:0/2373485226 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f340a55d0 0x7f1f3414a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:07.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.300+0000 7f1f3996d640 1 -- 172.21.15.29:0/2373485226 >> 172.21.15.29:0/2373485226 conn(0x7f1f340ab3f0 msgr2=0x7f1f341472d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:07.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.300+0000 7f1f1bfff640 4 mgrc handle_mgr_map Got map version 16 2024-04-25T19:46:07.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.300+0000 7f1f1bfff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] 2024-04-25T19:46:07.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.300+0000 7f1f1bfff640 4 mgrc reconnect waiting to retry connect until 602.582784s 2024-04-25T19:46:07.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.302+0000 7f1f3996d640 1 -- 172.21.15.29:0/2373485226 shutdown_connections 2024-04-25T19:46:07.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.302+0000 7f1f3996d640 1 -- 172.21.15.29:0/2373485226 wait complete. 2024-04-25T19:46:07.317 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T19:46:07.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.433+0000 7f4d53b4c640 1 Processor -- start 2024-04-25T19:46:07.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.433+0000 7f4d53b4c640 1 -- start start 2024-04-25T19:46:07.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.434+0000 7f4d53b4c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4d4c0aa9d0 0x7f4d4c0bbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:07.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.434+0000 7f4d53b4c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d4c0ad000 0x7f4d4c147250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:07.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.434+0000 7f4d53b4c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d4c0aade0 con 0x7f4d4c0acbf0 2024-04-25T19:46:07.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.434+0000 7f4d53b4c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f4d4c0ae6a0 con 0x7f4d4c0aa9d0 2024-04-25T19:46:07.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.434+0000 7f4d53b4c640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f4d4c14a0b0 con 0x7f4d4c0ad000 2024-04-25T19:46:07.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.434+0000 7f4d520e9640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d4c0ad000 0x7f4d4c147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:07.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.434+0000 7f4d518e8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4d4c0aa9d0 0x7f4d4c0bbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:07.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.434+0000 7f4d510e7640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4d4c0acbf0 0x7f4d4c146b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43308/0 (socket says 172.21.15.29:43308) 2024-04-25T19:46:07.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.434+0000 7f4d510e7640 1 -- 172.21.15.29:0/1822729212 learned_addr learned my addr 172.21.15.29:0/1822729212 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-25T19:46:07.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.435+0000 7f4d520e9640 1 -- 172.21.15.29:0/1822729212 >> v2:172.21.15.29:3300/0 conn(0x7f4d4c0aa9d0 msgr2=0x7f4d4c0bbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:46:07.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.435+0000 7f4d520e9640 1 --2- 172.21.15.29:0/1822729212 >> v2:172.21.15.29:3300/0 conn(0x7f4d4c0aa9d0 0x7f4d4c0bbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:07.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.435+0000 7f4d520e9640 1 -- 172.21.15.29:0/1822729212 >> v1:172.21.15.29:6789/0 conn(0x7f4d4c0acbf0 legacy=0x7f4d4c146b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:07.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.435+0000 7f4d520e9640 1 -- 172.21.15.29:0/1822729212 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d4c0ad780 con 0x7f4d4c0ad000 2024-04-25T19:46:07.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.435+0000 7f4d518e8640 1 --2- 172.21.15.29:0/1822729212 >> v2:172.21.15.29:3300/0 conn(0x7f4d4c0aa9d0 0x7f4d4c0bbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:46:07.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.435+0000 7f4d520e9640 1 --2- 172.21.15.29:0/1822729212 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d4c0ad000 0x7f4d4c147250 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4d44009d90 tx=0x7f4d4402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74910ec86c809f45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:07.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.436+0000 7f4d508e6640 1 -- 172.21.15.29:0/1822729212 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d44039070 con 0x7f4d4c0ad000 2024-04-25T19:46:07.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.436+0000 7f4d508e6640 1 -- 172.21.15.29:0/1822729212 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4d4402cc50 con 0x7f4d4c0ad000 2024-04-25T19:46:07.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.436+0000 7f4d508e6640 1 -- 172.21.15.29:0/1822729212 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d44034440 con 0x7f4d4c0ad000 2024-04-25T19:46:07.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.436+0000 7f4d53b4c640 1 -- 172.21.15.29:0/1822729212 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d4c0ad000 msgr2=0x7f4d4c147250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:07.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.436+0000 7f4d53b4c640 1 --2- 172.21.15.29:0/1822729212 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d4c0ad000 0x7f4d4c147250 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4d44009d90 tx=0x7f4d4402bea0 comp rx=0 tx=0).stop 2024-04-25T19:46:07.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.436+0000 7f4d53b4c640 1 -- 172.21.15.29:0/1822729212 shutdown_connections 2024-04-25T19:46:07.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.436+0000 7f4d53b4c640 1 --2- 172.21.15.29:0/1822729212 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d4c0ad000 0x7f4d4c147250 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:07.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.436+0000 7f4d53b4c640 1 --2- 172.21.15.29:0/1822729212 >> v2:172.21.15.29:3300/0 conn(0x7f4d4c0aa9d0 0x7f4d4c0bbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:07.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.436+0000 7f4d53b4c640 1 -- 172.21.15.29:0/1822729212 >> 172.21.15.29:0/1822729212 conn(0x7f4d4c073cc0 msgr2=0x7f4d4c0740d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:07.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.436+0000 7f4d53b4c640 1 -- 172.21.15.29:0/1822729212 shutdown_connections 2024-04-25T19:46:07.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.437+0000 7f4d53b4c640 1 -- 172.21.15.29:0/1822729212 wait complete. 2024-04-25T19:46:07.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.438+0000 7f4d53b4c640 1 Processor -- start 2024-04-25T19:46:07.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.438+0000 7f4d53b4c640 1 -- start start 2024-04-25T19:46:07.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.439+0000 7f4d53b4c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d4c0aa9d0 0x7f4d4c0b4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:07.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.439+0000 7f4d53b4c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d4c0acbf0 0x7f4d4c147130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:07.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.439+0000 7f4d53b4c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d4c0ad000 0x7f4d4c1e0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:07.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.439+0000 7f4d53b4c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4c0ae0a0 con 0x7f4d4c0acbf0 2024-04-25T19:46:07.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.439+0000 7f4d53b4c640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4c0bc4d0 con 0x7f4d4c0ad000 2024-04-25T19:46:07.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.439+0000 7f4d53b4c640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 -- 0x7f4d4c0ae220 con 0x7f4d4c0aa9d0 2024-04-25T19:46:07.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.439+0000 7f4d518e8640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d4c0aa9d0 0x7f4d4c0b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:07.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.439+0000 7f4d510e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d4c0acbf0 0x7f4d4c147130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.439+0000 7f4d518e8640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d4c0aa9d0 0x7f4d4c0b4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:50076/0 (socket says 172.21.15.29:50076) 2024-04-25T19:46:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.439+0000 7f4d510e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d4c0acbf0 0x7f4d4c147130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51352/0 (socket says 172.21.15.29:51352) 2024-04-25T19:46:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.439+0000 7f4d518e8640 1 -- 172.21.15.29:0/4238976164 learned_addr learned my addr 172.21.15.29:0/4238976164 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-25T19:46:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.439+0000 7f4d518e8640 1 -- 172.21.15.29:0/4238976164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d4c0ad000 msgr2=0x7f4d4c1e0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.439+0000 7f4d518e8640 1 --2- 172.21.15.29:0/4238976164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d4c0ad000 0x7f4d4c1e0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.439+0000 7f4d518e8640 1 -- 172.21.15.29:0/4238976164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d4c0acbf0 msgr2=0x7f4d4c147130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.439+0000 7f4d518e8640 1 --2- 172.21.15.29:0/4238976164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d4c0acbf0 0x7f4d4c147130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.439+0000 7f4d518e8640 1 -- 172.21.15.29:0/4238976164 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d4c0b4b40 con 0x7f4d4c0aa9d0 2024-04-25T19:46:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.439+0000 7f4d510e7640 1 --2- 172.21.15.29:0/4238976164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d4c0acbf0 0x7f4d4c147130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:46:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.439+0000 7f4d518e8640 1 --2- 172.21.15.29:0/4238976164 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d4c0aa9d0 0x7f4d4c0b4140 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4d4800e970 tx=0x7f4d4800ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.440+0000 7f4d427fc640 1 -- 172.21.15.29:0/4238976164 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d48017070 con 0x7f4d4c0aa9d0 2024-04-25T19:46:07.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.440+0000 7f4d427fc640 1 -- 172.21.15.29:0/4238976164 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4d4800ca00 con 0x7f4d4c0aa9d0 2024-04-25T19:46:07.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.440+0000 7f4d53b4c640 1 -- 172.21.15.29:0/4238976164 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d4c1e09b0 con 0x7f4d4c0aa9d0 2024-04-25T19:46:07.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.440+0000 7f4d427fc640 1 -- 172.21.15.29:0/4238976164 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4800cdf0 con 0x7f4d4c0aa9d0 2024-04-25T19:46:07.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.440+0000 7f4d53b4c640 1 -- 172.21.15.29:0/4238976164 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d4c0b3850 con 0x7f4d4c0aa9d0 2024-04-25T19:46:07.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.441+0000 7f4d53b4c640 1 -- 172.21.15.29:0/4238976164 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d4c1e0f00 con 0x7f4d4c0aa9d0 2024-04-25T19:46:07.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.442+0000 7f4d427fc640 1 -- 172.21.15.29:0/4238976164 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 16) ==== 144961+0+0 (secure 0 0 0) 0x7f4d4801a070 con 0x7f4d4c0aa9d0 2024-04-25T19:46:07.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.442+0000 7f4d427fc640 20 mgrc handle_mgr_map mgrmap(e 16) 2024-04-25T19:46:07.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.443+0000 7f4d427fc640 4 mgrc handle_mgr_map Got map version 16 2024-04-25T19:46:07.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.443+0000 7f4d427fc640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] 2024-04-25T19:46:07.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.443+0000 7f4d427fc640 4 mgrc reconnect Starting new session with [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] 2024-04-25T19:46:07.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.443+0000 7f4d427fc640 1 --2- 172.21.15.29:0/4238976164 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f4d300ac680 0x7f4d300aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:46:07.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.443+0000 7f4d427fc640 1 -- 172.21.15.29:0/4238976164 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(33..33 src has 1..33) ==== 4846+0+0 (secure 0 0 0) 0x7f4d480d96a0 con 0x7f4d4c0aa9d0 2024-04-25T19:46:07.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.443+0000 7f4d510e7640 1 --2- 172.21.15.29:0/4238976164 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f4d300ac680 0x7f4d300aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:46:07.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.443+0000 7f4d510e7640 1 --2- 172.21.15.29:0/4238976164 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f4d300ac680 0x7f4d300aeb90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c0b19d0 tx=0x7f4d3c009040 comp rx=0 tx=0).ready entity=mgr.4595 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:46:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.447+0000 7f4d427fc640 1 -- 172.21.15.29:0/4238976164 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4d480a4000 con 0x7f4d4c0aa9d0 2024-04-25T19:46:07.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:07.630+0000 7f4d53b4c640 1 -- 172.21.15.29:0/4238976164 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4d4c0507b0 con 0x7f4d4c0aa9d0 2024-04-25T19:46:08.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:08.299+0000 7f4d427fc640 1 -- 172.21.15.29:0/4238976164 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v34) ==== 169+0+0 (secure 0 0 0) 0x7f4d480a3900 con 0x7f4d4c0aa9d0 2024-04-25T19:46:08.300 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2024-04-25T19:46:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:08.304+0000 7f4d53b4c640 10 mgrc shutdown 2024-04-25T19:46:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:08.304+0000 7f4d53b4c640 1 -- 172.21.15.29:0/4238976164 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f4d300ac680 msgr2=0x7f4d300aeb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:08.304+0000 7f4d53b4c640 1 --2- 172.21.15.29:0/4238976164 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f4d300ac680 0x7f4d300aeb90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c0b19d0 tx=0x7f4d3c009040 comp rx=0 tx=0).stop 2024-04-25T19:46:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:08.304+0000 7f4d53b4c640 1 -- 172.21.15.29:0/4238976164 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d4c0aa9d0 msgr2=0x7f4d4c0b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:46:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:08.304+0000 7f4d53b4c640 1 --2- 172.21.15.29:0/4238976164 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d4c0aa9d0 0x7f4d4c0b4140 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4d4800e970 tx=0x7f4d4800ee30 comp rx=0 tx=0).stop 2024-04-25T19:46:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:08.304+0000 7f4d53b4c640 1 -- 172.21.15.29:0/4238976164 shutdown_connections 2024-04-25T19:46:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:08.304+0000 7f4d53b4c640 1 --2- 172.21.15.29:0/4238976164 >> [v2:172.21.15.102:6824/2959738929,v1:172.21.15.102:6825/2959738929] conn(0x7f4d300ac680 0x7f4d300aeb90 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:08.304+0000 7f4d53b4c640 1 --2- 172.21.15.29:0/4238976164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d4c0ad000 0x7f4d4c1e0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:08.304+0000 7f4d53b4c640 1 --2- 172.21.15.29:0/4238976164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d4c0acbf0 0x7f4d4c147130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:08.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:08.304+0000 7f4d53b4c640 1 --2- 172.21.15.29:0/4238976164 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4d4c0aa9d0 0x7f4d4c0b4140 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:46:08.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:08.304+0000 7f4d53b4c640 1 -- 172.21.15.29:0/4238976164 >> 172.21.15.29:0/4238976164 conn(0x7f4d4c073cc0 msgr2=0x7f4d4c0740d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:46:08.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:08.305+0000 7f4d53b4c640 1 -- 172.21.15.29:0/4238976164 shutdown_connections 2024-04-25T19:46:08.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-25T19:46:08.305+0000 7f4d53b4c640 1 -- 172.21.15.29:0/4238976164 wait complete. 2024-04-25T19:46:08.322 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-25T19:46:08.322 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T19:46:08.322 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:46:08.332 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T19:46:08.332 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T19:46:08.332 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:46:08.387 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T19:46:08.387 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T19:46:08.387 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:46:08.398 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T19:46:08.398 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-25T19:46:08.399 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-25T19:46:08.399 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:46:08.461 INFO:tasks.ceph.osd.0:Stopped 2024-04-25T19:46:08.462 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-25T19:46:08.462 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:46:08.495 INFO:tasks.ceph.osd.1:Stopped 2024-04-25T19:46:08.495 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-25T19:46:08.495 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:46:08.568 INFO:tasks.ceph.osd.2:Stopped 2024-04-25T19:46:08.569 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-25T19:46:08.569 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:46:08.640 INFO:tasks.ceph.osd.3:Stopped 2024-04-25T19:46:08.640 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-25T19:46:08.640 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-25T19:46:08.640 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:46:08.670 INFO:tasks.ceph.mgr.x:Stopped 2024-04-25T19:46:08.670 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-25T19:46:08.670 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:46:08.713 INFO:tasks.ceph.mgr.y:Stopped 2024-04-25T19:46:08.713 DEBUG:tasks.ceph.mgr.z:waiting for process to exit 2024-04-25T19:46:08.713 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:46:08.783 INFO:tasks.ceph.mgr.z:Stopped 2024-04-25T19:46:08.783 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-25T19:46:08.783 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-25T19:46:08.783 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:46:08.783 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.z is failed for ~0s 2024-04-25T19:46:08.797 INFO:tasks.ceph.mon.a:Stopped 2024-04-25T19:46:08.797 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-25T19:46:08.798 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:46:08.852 INFO:tasks.ceph.mon.c:Stopped 2024-04-25T19:46:08.852 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-25T19:46:08.852 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:46:08.864 INFO:tasks.ceph.mon.b:Stopped 2024-04-25T19:46:08.864 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-25T19:46:08.864 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_' | egrep -v 'replacing it with standby' | egrep -v 'No standby daemons available' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-25T19:46:08.923 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi029.front.sepia.ceph.com 2024-04-25T19:46:08.923 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-25T19:46:09.325 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T19:46:09.325 DEBUG:teuthology.orchestra.run.smithi102:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-25T19:46:09.757 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T19:46:09.757 DEBUG:teuthology.orchestra.run.smithi102:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-25T19:46:09.820 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T19:46:09.821 DEBUG:teuthology.orchestra.run.smithi102:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-25T19:46:09.872 INFO:tasks.ceph:Archiving mon data... 2024-04-25T19:46:09.873 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-04-25_11:25:24-rados:mgr-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673402/data/mon.a.tgz 2024-04-25T19:46:09.873 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-04-25T19:46:09.892 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.F9jsvzsjmw 2024-04-25T19:46:09.892 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.F9jsvzsjmw 2024-04-25T19:46:10.135 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.F9jsvzsjmw 2024-04-25T19:46:10.219 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.F9jsvzsjmw is 961KB 2024-04-25T19:46:10.328 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.F9jsvzsjmw 2024-04-25T19:46:10.343 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-04-25_11:25:24-rados:mgr-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673402/data/mon.c.tgz 2024-04-25T19:46:10.344 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-04-25T19:46:10.398 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.jdc1Qyl3fb 2024-04-25T19:46:10.398 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.jdc1Qyl3fb 2024-04-25T19:46:10.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.jdc1Qyl3fb 2024-04-25T19:46:10.778 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.jdc1Qyl3fb is 1MB 2024-04-25T19:46:10.891 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.jdc1Qyl3fb 2024-04-25T19:46:10.907 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-04-25_11:25:24-rados:mgr-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673402/data/mon.b.tgz 2024-04-25T19:46:10.907 DEBUG:teuthology.orchestra.run.smithi102:> mktemp 2024-04-25T19:46:10.924 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/tmp.xhUR0cBkxX 2024-04-25T19:46:10.924 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.xhUR0cBkxX 2024-04-25T19:46:11.218 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0666 /tmp/tmp.xhUR0cBkxX 2024-04-25T19:46:11.300 DEBUG:teuthology.orchestra.remote:smithi102:/tmp/tmp.xhUR0cBkxX is 1MB 2024-04-25T19:46:11.412 DEBUG:teuthology.orchestra.run.smithi102:> rm -fr /tmp/tmp.xhUR0cBkxX 2024-04-25T19:46:11.427 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-25T19:46:11.427 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T19:46:11.430 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T19:46:11.510 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T19:46:11.544 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T19:46:11.544 INFO:tasks.ceph:Archiving crash dumps... 2024-04-25T19:46:11.546 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-04-25_11:25:24-rados:mgr-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673402/remote/smithi029/crash 2024-04-25T19:46:11.547 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T19:46:11.577 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-04-25_11:25:24-rados:mgr-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673402/remote/smithi102/crash 2024-04-25T19:46:11.578 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T19:46:11.607 INFO:tasks.ceph:Shutting down logrotate 2024-04-25T19:46:11.607 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:46:11.621 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:46:11.692 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-25T19:46:11.717 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-25T19:46:11.756 INFO:tasks.ceph:Compressing logs... 2024-04-25T19:46:11.756 DEBUG:teuthology.orchestra.run.smithi029:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T19:46:11.760 DEBUG:teuthology.orchestra.run.smithi102:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T19:46:11.782 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32772.log 2024-04-25T19:46:11.782 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32879.log 2024-04-25T19:46:11.812 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31492.log 2024-04-25T19:46:11.812 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30760.log 2024-04-25T19:46:11.812 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29937.log 2024-04-25T19:46:11.813 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32104.log 2024-04-25T19:46:11.813 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30365.log 2024-04-25T19:46:11.813 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32879.log: /var/log/ceph/ceph-client.admin.31492.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32772.log.gz 2024-04-25T19:46:11.813 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30760.log: /var/log/ceph/ceph-client.admin.29937.log: gzip/var/log/ceph/ceph-client.admin.32104.log: -5 --verbose -- 88.4% 88.1% /var/log/ceph/ceph-client.admin.30323.log 2024-04-25T19:46:11.813 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31492.log.gz -- replaced with /var/log/ceph/ceph-client.admin.32879.log.gz 2024-04-25T19:46:11.813 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-04-25T19:46:11.813 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.29937.log.gz 2024-04-25T19:46:11.813 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.30760.log.gz 2024-04-25T19:46:11.813 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30365.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.32104.log.gz 2024-04-25T19:46:11.813 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30844.log 2024-04-25T19:46:11.813 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.30365.log.gz 2024-04-25T19:46:11.813 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30323.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T19:46:11.813 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2%/var/log/ceph/ceph-client.admin.30844.log: -- replaced with /var/log/ceph/ceph-client.admin.30323.log.gz 2024-04-25T19:46:11.813 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33333.log 2024-04-25T19:46:11.814 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30844.log.gz 2024-04-25T19:46:11.814 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31493.log 2024-04-25T19:46:11.814 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32658.log 2024-04-25T19:46:11.814 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.33333.log.gz 2024-04-25T19:46:11.814 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30672.log 2024-04-25T19:46:11.814 INFO:teuthology.orchestra.run.smithi029.stderr: 91.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T19:46:11.814 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31493.log.gz 2024-04-25T19:46:11.814 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31894.log 2024-04-25T19:46:11.814 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33062.log 2024-04-25T19:46:11.814 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32658.log.gz 2024-04-25T19:46:11.814 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30672.log.gz 2024-04-25T19:46:11.814 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33146.log 2024-04-25T19:46:11.814 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.31894.log.gz 2024-04-25T19:46:11.814 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33062.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T19:46:11.815 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30579.log 2024-04-25T19:46:11.815 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33062.log.gz 2024-04-25T19:46:11.815 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- 88.5% /var/log/ceph/ceph-client.admin.32314.log -- replaced with /var/log/ceph/ceph-client.admin.33146.log.gz 2024-04-25T19:46:11.815 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-04-25T19:46:11.815 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30449.log 2024-04-25T19:46:11.815 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32314.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.30579.log.gz 2024-04-25T19:46:11.815 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31432.log 2024-04-25T19:46:11.815 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31264.log 2024-04-25T19:46:11.815 INFO:teuthology.orchestra.run.smithi029.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.32314.log.gz 2024-04-25T19:46:11.815 INFO:teuthology.orchestra.run.smithi029.stderr: 94.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T19:46:11.815 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31432.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.30449.log.gz 2024-04-25T19:46:11.815 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32146.log 2024-04-25T19:46:11.815 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5%/var/log/ceph/ceph-client.admin.31264.log: -- replaced with /var/log/ceph/ceph-client.admin.31432.log.gz 2024-04-25T19:46:11.815 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30407.log 2024-04-25T19:46:11.815 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31264.log.gz 2024-04-25T19:46:11.815 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30535.log 2024-04-25T19:46:11.816 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32146.log.gz 2024-04-25T19:46:11.816 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32610.log 2024-04-25T19:46:11.816 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31810.log 2024-04-25T19:46:11.816 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30407.log.gz 2024-04-25T19:46:11.816 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32610.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.30535.log.gz 2024-04-25T19:46:11.816 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32717.log 2024-04-25T19:46:11.816 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32610.log.gz 2024-04-25T19:46:11.816 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31322.log 2024-04-25T19:46:11.816 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32717.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31810.log.gz -5 2024-04-25T19:46:11.816 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.32927.log 2024-04-25T19:46:11.816 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32717.log.gz 2024-04-25T19:46:11.816 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31322.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-25T19:46:11.816 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.31322.log.gz 2024-04-25T19:46:11.817 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32747.log 2024-04-25T19:46:11.817 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32927.log: /var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26348.log 2024-04-25T19:46:11.817 INFO:teuthology.orchestra.run.smithi029.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32927.log.gz 2024-04-25T19:46:11.817 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31642.log 2024-04-25T19:46:11.817 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32747.log.gz 2024-04-25T19:46:11.817 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.tmp-client.admin.26348.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-25T19:46:11.817 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.26348.log.gz 2024-04-25T19:46:11.817 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30630.log 2024-04-25T19:46:11.817 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31155.log 2024-04-25T19:46:11.817 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.31642.log.gz 2024-04-25T19:46:11.817 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28679.log 2024-04-25T19:46:11.817 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.30630.log.gz/var/log/ceph/ceph-client.admin.31155.log: 2024-04-25T19:46:11.817 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-25T19:46:11.817 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31155.log.gz 2024-04-25T19:46:11.817 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31452.log 2024-04-25T19:46:11.818 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.28679.log: /var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31156.log 2024-04-25T19:46:11.818 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28679.log.gz 2024-04-25T19:46:11.818 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31452.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-25T19:46:11.818 INFO:teuthology.orchestra.run.smithi029.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-25T19:46:11.818 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31452.log.gz 2024-04-25T19:46:11.818 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32634.log 2024-04-25T19:46:11.818 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.31156.log.gz 2024-04-25T19:46:11.818 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.30115.log 2024-04-25T19:46:11.818 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.32634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28726.log 2024-04-25T19:46:11.818 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30279.log 2024-04-25T19:46:11.818 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32634.log.gz 2024-04-25T19:46:11.818 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30714.log 2024-04-25T19:46:11.818 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.28726.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30115.log.gz 2024-04-25T19:46:11.818 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30279.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.28726.log.gz 2024-04-25T19:46:11.818 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-25T19:46:11.819 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33188.log 2024-04-25T19:46:11.819 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30714.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.30279.log.gz 2024-04-25T19:46:11.819 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32272.log 2024-04-25T19:46:11.819 INFO:teuthology.orchestra.run.smithi029.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.30714.log.gz 2024-04-25T19:46:11.819 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32903.log 2024-04-25T19:46:11.819 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32272.log: gzip -5 --verbose 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33188.log.gz 2024-04-25T19:46:11.819 INFO:teuthology.orchestra.run.smithi029.stderr: -- /var/log/ceph/ceph-client.admin.32951.log 2024-04-25T19:46:11.819 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32903.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32272.log.gz 2024-04-25T19:46:11.819 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33375.log 2024-04-25T19:46:11.819 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32903.log.gz 2024-04-25T19:46:11.819 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32062.log 2024-04-25T19:46:11.819 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33375.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32951.log.gz 2024-04-25T19:46:11.819 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31978.log 2024-04-25T19:46:11.819 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33375.log.gz 2024-04-25T19:46:11.819 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31054.log 2024-04-25T19:46:11.820 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32062.log: /var/log/ceph/ceph-client.admin.31978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31768.log 2024-04-25T19:46:11.820 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32062.log.gz 2024-04-25T19:46:11.820 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31323.log 2024-04-25T19:46:11.820 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.31978.log.gz 2024-04-25T19:46:11.820 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28495.log 2024-04-25T19:46:11.820 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31054.log.gz 2024-04-25T19:46:11.820 INFO:teuthology.orchestra.run.smithi029.stderr:gzip 88.7% -5 --verbose -- /var/log/ceph/ceph-client.admin.33272.log 2024-04-25T19:46:11.820 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31323.log: -- replaced with /var/log/ceph/ceph-client.admin.31768.log.gz 2024-04-25T19:46:11.820 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.28495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29854.log 2024-04-25T19:46:11.820 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31323.log.gz 2024-04-25T19:46:11.820 INFO:teuthology.orchestra.run.smithi029.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.28495.log.gz 2024-04-25T19:46:11.820 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30928.log 2024-04-25T19:46:11.820 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33272.log.gz 2024-04-25T19:46:11.820 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33230.log 2024-04-25T19:46:11.820 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29854.log.gz 2024-04-25T19:46:11.821 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33104.log 2024-04-25T19:46:11.821 INFO:teuthology.orchestra.run.smithi029.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-25T19:46:11.821 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33230.log: gzip 87.7% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.30928.log.gz -- 2024-04-25T19:46:11.821 INFO:teuthology.orchestra.run.smithi029.stderr: /var/log/ceph/ceph-client.admin.31684.log 2024-04-25T19:46:11.821 INFO:teuthology.orchestra.run.smithi029.stderr:gzip/var/log/ceph/ceph-client.admin.33104.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30802.log 2024-04-25T19:46:11.821 INFO:teuthology.orchestra.run.smithi029.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.33230.log.gz 2024-04-25T19:46:11.821 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32230.log 2024-04-25T19:46:11.821 INFO:teuthology.orchestra.run.smithi029.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.33104.log.gz 2024-04-25T19:46:11.821 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30802.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31684.log.gz 2024-04-25T19:46:11.821 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30886.log 2024-04-25T19:46:11.821 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30802.log.gz 2024-04-25T19:46:11.821 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30237.log 2024-04-25T19:46:11.821 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32230.log.gz 2024-04-25T19:46:11.821 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29993.log 2024-04-25T19:46:11.821 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30491.log 2024-04-25T19:46:11.821 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30237.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.30886.log.gz 2024-04-25T19:46:11.822 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31154.log 2024-04-25T19:46:11.822 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30237.log.gz 2024-04-25T19:46:11.822 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29993.log.gz 2024-04-25T19:46:11.822 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30970.log 2024-04-25T19:46:11.822 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31324.log 2024-04-25T19:46:11.822 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30491.log.gz 2024-04-25T19:46:11.822 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30970.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.31154.log.gz 2024-04-25T19:46:11.822 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30035.log 2024-04-25T19:46:11.822 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32854.log 2024-04-25T19:46:11.822 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31324.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.30970.log.gz 2024-04-25T19:46:11.822 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30195.log 2024-04-25T19:46:11.822 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31324.log.gz 2024-04-25T19:46:11.822 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30035.log.gz 2024-04-25T19:46:11.822 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32188.log 2024-04-25T19:46:11.823 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32854.log: /var/log/ceph/ceph-client.admin.30195.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-25T19:46:11.823 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32854.log.gz 2024-04-25T19:46:11.823 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32188.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31600.log 2024-04-25T19:46:11.823 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30195.log.gz 2024-04-25T19:46:11.823 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32188.log.gz 2024-04-25T19:46:11.823 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31852.log 2024-04-25T19:46:11.823 INFO:teuthology.orchestra.run.smithi029.stderr: 89.6% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-25T19:46:11.823 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31153.log 2024-04-25T19:46:11.823 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31600.log: /var/log/ceph/ceph-client.admin.31852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31936.log 2024-04-25T19:46:11.823 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31600.log.gz 2024-04-25T19:46:11.823 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31153.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31852.log.gz 2024-04-25T19:46:11.823 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31012.log 2024-04-25T19:46:11.823 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32020.log 2024-04-25T19:46:11.823 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31936.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31153.log.gz 2024-04-25T19:46:11.823 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31726.log 2024-04-25T19:46:11.823 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.31936.log.gz 2024-04-25T19:46:11.824 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32020.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31012.log.gz 2024-04-25T19:46:11.824 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31726.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32020.log.gz 2024-04-25T19:46:11.824 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31726.log.gz 2024-04-25T19:46:11.828 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T19:46:11.828 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-25T19:46:11.828 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30849.log 2024-04-25T19:46:11.828 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T19:46:11.828 INFO:teuthology.orchestra.run.smithi102.stderr: 84.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T19:46:11.828 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.30849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30809.log 2024-04-25T19:46:11.828 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph.audit.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30849.log.gz 2024-04-25T19:46:11.828 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30889.log 2024-04-25T19:46:11.828 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.30809.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-25T19:46:11.828 INFO:teuthology.orchestra.run.smithi102.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.30809.log.gz 2024-04-25T19:46:11.829 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.30889.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-25T19:46:11.829 INFO:teuthology.orchestra.run.smithi102.stderr: 89.7% 86.7% -- replaced with /var/log/ceph/ceph-client.admin.30889.log.gz -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T19:46:11.829 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-04-25T19:46:11.829 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-25T19:46:11.829 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-mds.b.log: /var/log/ceph/ceph-osd.3.log: 89.7% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-25T19:46:11.829 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-25T19:46:11.829 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.z.log 2024-04-25T19:46:11.829 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30929.log 2024-04-25T19:46:11.829 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-25T19:46:11.829 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.30929.log: /var/log/ceph/ceph-mon.b.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.30929.log.gz 2024-04-25T19:46:11.835 INFO:teuthology.orchestra.run.smithi102.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.z.log.gz 2024-04-25T19:46:11.864 INFO:teuthology.orchestra.run.smithi102.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-25T19:46:11.865 INFO:teuthology.orchestra.run.smithi029.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-25T19:46:11.921 INFO:teuthology.orchestra.run.smithi102.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-25T19:46:11.922 INFO:teuthology.orchestra.run.smithi029.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-25T19:46:12.525 INFO:teuthology.orchestra.run.smithi102.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-25T19:46:12.529 INFO:teuthology.orchestra.run.smithi102.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-25T19:46:12.536 INFO:teuthology.orchestra.run.smithi102.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-25T19:46:12.539 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-04-25T19:46:12.539 INFO:teuthology.orchestra.run.smithi102.stderr:real 0m0.724s 2024-04-25T19:46:12.539 INFO:teuthology.orchestra.run.smithi102.stderr:user 0m2.182s 2024-04-25T19:46:12.539 INFO:teuthology.orchestra.run.smithi102.stderr:sys 0m0.103s 2024-04-25T19:46:12.618 INFO:teuthology.orchestra.run.smithi029.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-25T19:46:12.619 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-04-25T19:46:12.620 INFO:teuthology.orchestra.run.smithi029.stderr:real 0m0.847s 2024-04-25T19:46:12.620 INFO:teuthology.orchestra.run.smithi029.stderr:user 0m1.050s 2024-04-25T19:46:12.620 INFO:teuthology.orchestra.run.smithi029.stderr:sys 0m0.088s 2024-04-25T19:46:12.620 INFO:tasks.ceph:Archiving logs... 2024-04-25T19:46:12.620 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_11:25:24-rados:mgr-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673402/remote/smithi029/log 2024-04-25T19:46:12.621 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T19:46:12.894 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_11:25:24-rados:mgr-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673402/remote/smithi102/log 2024-04-25T19:46:12.895 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T19:46:13.228 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T19:46:13.238 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T19:46:13.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T19:46:13.241 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T19:46:13.327 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T19:46:13.338 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T19:46:13.338 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T19:46:13.341 DEBUG:teuthology.orchestra.run.smithi102:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T19:46:13.354 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-04-25T19:46:13.354 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-04-25T19:46:13.358 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T19:46:13.358 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-04-25T19:46:13.358 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv01.front.sepia.ceph.com 3 6 377 48 +1053us[+1043us] +/- 49ms 2024-04-25T19:46:13.358 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 48 -1171us[-1171us] +/- 55ms 2024-04-25T19:46:13.358 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 -6297ns[ -17us] +/- 80ms 2024-04-25T19:46:13.358 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:46:13.359 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T19:46:13.360 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-04-25T19:46:13.360 INFO:teuthology.orchestra.run.smithi102.stdout:^* hv01.front.sepia.ceph.com 3 6 377 49 +1049us[+1051us] +/- 49ms 2024-04-25T19:46:13.360 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 49 -1165us[-1162us] +/- 55ms 2024-04-25T19:46:13.360 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 48 -3670ns[-3670ns] +/- 80ms 2024-04-25T19:46:13.360 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:46:13.360 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T19:46:13.370 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T19:46:13.370 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T19:46:13.382 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T19:46:13.464 DEBUG:teuthology.orchestra.run.smithi102:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T19:46:13.520 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T19:46:13.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:46:13.548 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T19:46:13.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:46:13.574 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T19:46:13.586 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T19:46:13.595 INFO:teuthology.task.internal:Duration was 531.188351 seconds 2024-04-25T19:46:13.596 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T19:46:13.605 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T19:46:13.606 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T19:46:13.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T19:46:13.645 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T19:46:13.650 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T19:46:13.784 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.z has been restored 2024-04-25T19:46:14.057 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T19:46:14.057 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2024-04-25T19:46:14.057 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T19:46:14.118 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T19:46:14.118 DEBUG:teuthology.orchestra.run.smithi102:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T19:46:14.144 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T19:46:14.145 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T19:46:14.161 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T19:46:14.211 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T19:46:14.211 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T19:46:14.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T19:46:14.345 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T19:46:14.355 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T19:46:14.355 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T19:46:14.379 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T19:46:14.402 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T19:46:14.413 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T19:46:14.422 DEBUG:teuthology.orchestra.run.smithi102:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T19:46:14.442 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2024-04-25T19:46:14.464 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = core 2024-04-25T19:46:14.494 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T19:46:14.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:46:14.510 DEBUG:teuthology.orchestra.run.smithi102:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T19:46:14.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:46:14.548 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T19:46:14.562 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T19:46:14.563 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_11:25:24-rados:mgr-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673402/remote/smithi029 2024-04-25T19:46:14.563 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T19:46:14.596 DEBUG:teuthology.misc:Transferring archived files from smithi102:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_11:25:24-rados:mgr-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673402/remote/smithi102 2024-04-25T19:46:14.596 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T19:46:14.627 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T19:46:14.627 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T19:46:14.633 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T19:46:14.676 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T19:46:14.688 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T19:46:14.689 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T19:46:14.698 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T19:46:14.698 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T19:46:14.701 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T19:46:14.713 INFO:teuthology.orchestra.run.smithi029.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 19:46 /home/ubuntu/cephtest 2024-04-25T19:46:14.730 INFO:teuthology.orchestra.run.smithi102.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 19:46 /home/ubuntu/cephtest 2024-04-25T19:46:14.731 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T19:46:14.743 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T19:46:14.767 INFO:teuthology.run:Summary data: description: rados:mgr/{clusters/{2-node-mgr} debug/mgr distro/{centos_latest} mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-hybrid} tasks/{1-install 2-ceph 3-mgrmodules 4-units/workunits}} duration: 531.1883509159088 flavor: default owner: scheduled_pdonnell@teuthology success: true 2024-04-25T19:46:14.767 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T19:46:14.852 INFO:teuthology.run:pass